Installation

• Install Quickstart VDK
  • Prerequisites
  • Installation
• Install VDK Plugins
• Install Versatile Data Kit Control Service
  • Install locally
  • Install in Production

Install Quickstart VDK

Versatile Data Kit SDK (VDK) is the go to tool for developing and running Data Jobs locally.

Prerequisites

Versatile Data Kit CLI requires Python 3.7+. If you're new to Python, we recommend Anaconda.

It's recommended to have the latest version of pip :

pip install -U pip setuptools wheel

Installation

pip install quickstart-vdk

This will install VDK with support for some common databases and job lifecycle management operations.

In order to upgrade use:

pip install --upgrade --upgrade-strategy eager quickstart-vdk

See help to see what you can do:

vdk --help

Check out Getting Started to create your first Data Job and the Examples for the various things you can do with Versatile Data Kit.

Install VDK Plugins

VDK comes with a number of plugins that can be installed to change or extend its behavior.

pip install <vdk-plugin-name>

Once the plugin is installed, vdk automatically finds it and activates it. Check the plugin documentation for options. Run vdk config-help for a list of all configuration settings of all plugins.

You can find a list of plugins that we have already developed in plugins directory.

See the plugins README for more info.

Install Versatile Data Kit Control Service

Versatile Data Kit includes Control Service server enabling deploying, managing and monitoring Data Jobs. You can see the REST API Swagger documentation here

Install locally

Prerequisites

• Install helm
• Install docker
• Install kind (version 0.11.1 or later)

Then run:

   vdk server --install 

This will install Control Service on your local machine and configure local installation of VDK to use it.

Install in Production

In production, use the helm chart to install and configure it.

Prerequisites

• Install helm
• Kubernetes

Then run

helm repo add vdk-gitlab https://gitlab.com/api/v4/projects/28814611/packages/helm/stable
helm install my-release vdk-gitlab/pipelines-control-service 

For full production installation, it's recommended to follow this example

Read more about the installation in the Versatile Data Kit Control Service Chart Github repository

See list of all released versions of the helm chart here

Use

# see Help to see what you can do and start playing around
vdk --help 
# for example create a new job:
vdk create -u <URI-YOU-GOT-FROM-helm-install-OUTPUT>