Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Update import documentation. #555

Merged
merged 1 commit into from
Dec 8, 2015
Merged

Update import documentation. #555

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
73 changes: 56 additions & 17 deletions docs/developer/import.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,23 +6,30 @@ Thanks to Tom De Caluwé for his work on [Reaction Import](https://github.com/td

## Contents
- [Architecture](#architecture)
- [Batch inserts and fixture data](#batch-inserts-and-fixture-data)
- [Choosing a suitable key](#choosing-a-suitable-key)
- [Importing from files](#importing-from-files)
- [Automatic collection detection](#automatic-collection-detection)

- [Reference](#reference)
- [`ReactionImport.flush`](#flush)
- [`ReactionImport.fixture`](#fixture)
- [`ReactionImport.object`](#object)
- [`ReactionImport.process`](#process)
- [`ReactionImport.identify`](#identify)
- [`ReactionImport.load`](#load)
- [`ReactionImport.indication`](#indication)
- [`ReactionImport.shop`](#shop)
- [`ReactionImport.tag`](#tag)
- [`ReactionImport.product`](#product)
- [`ReactionImport.process`](#process)

- [Image uploader](#image-uploader)

## Architecture
The package only exports one global variable, `ReactionImport`, and is structured around an import buffer. This buffer is used to support fast bulk updates and inserts by sending all the queries at once to the database server. The import buffer can be filled with a number of webshop entities. Currently supported are:
- Tags
- Products
- Shops
- Packages
- Translations
- Tags
- Products

All the import functions have a similar signature:

Expand All @@ -34,8 +41,20 @@ The key is an object used to identify the product in the database (and is passed

When importing entities which need to be referenced by other entities it is recommended to provide a custom unique `_id` value as the `key`, thus avoiding an extra query to obtain this `_id`.

### Batch inserts and fixture data

By default documents are inserted with the MongoDB `$set` modifier. This however poses a problem when importing data which only needs to be imported if it's not already present in the database, ignoring changes made to the document after it was imported. To accomodate such situations, `ReactionImport` provides the `fixture()` modifier method. This method modifies `ReactionImport` to use the `$setOnInsert` modifier when sending documents to the database. It can be used as follows:

```javascript
ReactionImport.fixture().process(...);
```

This can also be used for doing batch inserts, with the restriction that a key has to provided for each imported document. Direct database inserts should be avoided since they can potentially create multiple copies of an import document (if the key provided isn't defined on the MongoDB collection as a unique key). Imports often need to handle a lot of data and should be stoppable and resumable. This is why `ReactionImport` avoids messing up the database by not supporting direct inserts.

### Choosing a suitable key
When importing documents that can be referenced by other documents, it might be a good idea to use the foreign key directly as the import key. Otherwise an extra query is needed to determine the foreign key, which in turn requires flushing the buffer.
In general, a chosen key should uniquely determine the imported object in the data source.

When importing a document together with some documents referencing it, try to use a foreign key available while importing the data. Specifically if the `_id` field is used as a foreign key, it might be a good idea to provide an `_id` field in the imported document. Otherwise an extra query is needed to consult the document in the database, which is not possible without flushing the import buffer.

### Importing from files
The package includes a helper function, [`ReactionImport.process`](#process), which accepts json arrays. It can be used to populate the database, importing some products might be as easy as:
Expand All @@ -44,9 +63,37 @@ The package includes a helper function, [`ReactionImport.process`](#process), wh
ReactionImport.process(Assets.getText(file), ['title'], ReactionImport.product);
```

### Automatic collection detection

The `ReactionImport.identify()` method is provided to determine the collection associated with a document. It is internally called by `ReactionImport.load()` which tries to load any document thorwn at it. These methods are mainly intented to be used with `ReactionImport.process()`, allowing json arrays mixing data from multiple collections to be imported.

## Reference
### `ReactionImport.flush()` <a name="flush"></a>
Read everything from the import buffer into the database.
### `ReactionImport.flush([collection])` <a name="flush"></a>
Send the contents of the import buffer to the database. Accepts an optional parameter restricting flushing data to a certain MongoDB collection.

### `ReactionImport.fixture()` <a name="fixture"></a>
Tell `ReactionImport` to use the `$setOnInsert` modifier when sending documents to MongoDB.

### `ReactionImport.object(collection, key, value)` <a name="object"></a>

Import a document with an associated key into a designated collection. Keep in mind that the other import methods specific to a certain collection usually only wrap this method, but can potentially extend it with extra logic. As an example, `ReactionImport.shop()` also loads translations associated with a given shop.

### `ReactionImport.process(json, key, callback)` <a name="process"></a>
Processes a json array with a given callback. The `key` argument should be used to pass an array containing the fields to be used as the import key.

The callback should accept two parameters: a key and the data to be imported. Usually this will be one of the `ReactionImport` entity import functions, although a custom callback can be used as well. A custom callback can, for example, be used to pass the parent key when importing products/variants.

### `ReactionImport.identify(value)` <a name="identify"></a>

Tries to associate a MongoDB collection with a certain document. Throws an error if it fails to do so or if results are ambiguous.

### `ReactionImport.load(key, value)` <a name="load"></a>

Try to import any document with an associated key, using `ReactionImport.identify()` to determine the collection.

### `ReactionImport.indication(field, collection, probability)` <a name="indication"></a>

Allow packages to extend the auto collection detection facility of `ReactionImport`. It is used to add a certain field as an indication that the document belongs to a certain collection. The probability should be a number between zero and one, where one is the strongest possible indication.

### `ReactionImport.tag(key, value)` <a name="tag"></a>
As explained above, for importing tags it might be interesting to provide a suitable `_id` as the key:
Expand All @@ -57,7 +104,7 @@ var tag;
ReactionImport.tag({ '_id': tag }, { 'name': tag, 'isTopLevel': true });
```

We can now directly use the `tag` value as a value in the `product.hashtags` array. If we didn't know the `_id` for our tag we'd need to commit the import buffer first and query for the tag with the key we provided, just to obtain this `_id`.
We can now directly use the `tag` value as a value in the `hashtags` array in the product schema. If we didn't know the `_id` for our tag we'd need to commit the import buffer first and query for the tag with the key we provided, only to obtain this `_id`.

### `Import.product(key, value, parent)` <a name="product"></a>
The product import provides an extra optional argument `parent` to accomodate for the product hierarchy structure planned for reaction. Currently this extra argument is used to import a variant where `key` identifies the variant and `parent` identifies the product. When called with two arguments the function will import a product. An example:
Expand All @@ -68,11 +115,3 @@ var title, description, tag, ean, price;
ReactionImport.product({ 'title': title }, { 'description': description, 'hashtags': [tag] });
ReactionImport.product({ 'ean': ean }, { 'price': price }, { 'title': title });
```

### `ReactionImport.process(json, key, callback)` <a name="process"></a>
Processes a json array with a given callback. The `key` argument should be used to pass an array containing the fields to be used as the import key.

The callback should accept two parameters: a key and the data to be imported. Usually this will be one of the `ReactionImport` entity import functions, although a custom callback can be used as well. A custom callback can, for example, be used to pass the parent key when importing products/variants.

## Image uploader
This package also includes a bulk image uploader as these often need to be uploaded separately when importing from an ERP solution. Currently a filename in the format `${product.variants.barcode}.[${priority}.]${extension}` is expected (where the priority is optional). Images will be used for the product if the priority is equal to zero, but will always show when the corresponding variant is consulted. This uploader can be accessed through the reaction dashboard.