processing data

title: Processing Data

Processing Data

tabel ORM offers two methods which you can use for batch-processing of data in your database. These methods are generally used when performing database wide modifications because of major changes in business requirements, or when syncing different data-sources. Detailed below:

---

.batchReduce

.batchReduce(batchSize=1000, batchReducer=((nextVal, rows) => {}), initialVal=null)

Example: Syncing data from relational-db to a nosql-db, like elastic-search

function syncPostsToElasticSearch() {
  // indexes posts with tags on elastic-search, 100 at a time.
  return table('posts').eagerLoad('tags').batchReduce(100, (promise, posts) => promise.then(() => {
    return Promise.all(posts.map((p) => indexPostOnElasticSearch(p)));
  }), Promise.resolve());
}

---

.reduce

.reduce(rowReducer=((nextVal, row) => {}), initialVal=null)

Internally, .reduce applies batch-reduce on batches of 1000 rows. However, only the reduce op of a single row is exposed to the developer.

Example: Adding a "type" field to the media-items attached to a post, based on the post.

function addTypeToMediaItems() {
  return table('posts').eagerLoad('mediaItems').reduce((promise, post) => promise.then(() => {
    return Promise.all(post.mediaItems.map((m) => {
      return getTypeByPostAndMediaItem(post, m)
        .then((type) => table('media_items').update(m.id, {type}))
      ;
    }));
  }), Promise.resolve());
}

---

Next > Caching Data