Skip to content

The primary module of Contentstack DataSync that syncs Contentstack data with your server.

License

Notifications You must be signed in to change notification settings

contentstack/datasync-manager

Repository files navigation

Contentstack

Contentstack is a headless CMS with an API-first approach. It is a CMS that developers can use to build powerful cross-platform applications in their favorite languages. Build your application frontend, and Contentstack will take care of the rest. Read More.

Contentstack DataSync Manager

Contentstack DataSync lets you sync your Contentstack data with your database, enabling you to save data locally and serve content directly from your database. It is a combination of four powerful modules that is DataSync Webhook Listener, DataSync Manager, DataSync Asset Store Filesystem, DataSync Content Store — Filesystem and MongoDB.

The DataSync Manager is one of the four important components of Contentstack DataSync. When any publish, unpublish, or delete operations are performed on assets or content, the DataSync Manager fetches the data and sends it to Content Store. It uses Contentstack's Sync APIs to sync data from Contentstack with your preferred database — Filesystem and MongoDB in our case.

Prerequisite

  • nodejs v8+

Usage

This is how DataSync Manager works with DataSync boilerplate:

const assetStore = require('@contentstack/datasync-asset-store-filesystem')
const contentStore = require('@contentstack/datasync-content-store-filesystem')
const listener = require('@contentstack/webhook-listener')
const syncManager = require('@contentstack/datasync-manager') // <--
const config = require('./config')

// Set asset, content store, listener and config to Datasync Manager
syncManager.setAssetStore(assetStore)
syncManager.setContentStore(contentStore)
syncManager.setListener(listener)
syncManager.setConfig(config)

// start DataSync manager
syncManager.start()
  .then(() => {
    console.log('Contentstack sync started successfully!')
  })
  .catch(console.error)

You can replace @contentstack/datasync-content-store-filesystem used above, with @contentstack/datasync-content-store-mongodb and switch content store databases.

Configuration

  • Here's a list of contentstack's configuration keys
Key Name Default Description
apiKey Required. Your stack's API key
deliveryToken Required. Your environment's delivery token
sync_token Token from where you'd like to start the process
pagination_token Token from where you'd like to start the process
MAX_RETRY_LIMIT 6 Number of times the API call would retry, if the server fails
  • Here's a list of configuration keys for contentstack datasync-manager:
Property Default Description
cooloff 3000 Number of ms the app would wait before making a subsequent Sync API request
enableAssetReferences true This is not implemented, you can help: #33
enableContentReferences true This is not implemented, you can help: #34
limit 100 Number of items fetched in each Sync API request made.
maxsize 2097152 Maximum file size of files (ex: .ledger file)
queue.pause_threshold 10000 The min-max internal queue size of contentstack-sync-manager. The DataSync manager would pause making Sync API requests once the internal queue size exceeds pause_threshold count.
queue.resume_threshold 5000 The min-max internal queue size of contentstack-sync-manager. The DataSync manager will resume Sync API requests once the queue size goes below resume_threshold count.
saveFilteredItems true When enabled, the app will log items that have been filtered out. Items can be filtered out due to the following:
  • Missing required keys
  • Language filters applied via config
filters
  • content_type_uid
  • locale
  • action
Filters allow you to narrow down the items that'd be returned by the SYNC API response. Ex: {filters: {content_type_uid: ['blogs']}, locale: ['en-us'], action: ['publish']}

And here's an example to get you started:

{
  // exmaple to override default values
  contentstack: {
      apiKey: '',
      deliveryToken: '',
      sync_token: '',
      MAX_RETRY_LIMIT: 5
    }
  },
  syncManager: {
      cooloff: 2000,
      limit: 80,
      filters: {
        content_type_uid: ['blogs'],
        locale: ['es-es', 'en-us'],
        action: ['publish', 'unpublish']
      }
    }
  }
}

Further Reading

Support and Feature requests

If you have any issues working with the library, please file an issue here at Github. You can send us an e-mail at [email protected] if you have any support or feature requests. Our support team is available 24/7 on the intercom. You can always get in touch and give us an opportunity to serve you better!

Licence

This repository is published under the MIT license.