datalake-kubernetes

Collection of docker images, helm charts and other tools needed to build DataLake on Kubernetes.

Usage

Install Monitoring Components

make install make install-monitoring

Delete Monitoring Components

make delete-monitoring

Install Base Charts

make install or upgrade

helm dependency build charts/datalake && helm install -n mylake charts/datalake --set tags.base=true --debug

helm dependency build charts/datalake && helm install -n mylake charts/datalake --set tags.base=false --set tags.alluxio=true --debug

Create tmp folder for PostgreSQL

export ANSIBLE_HOST_KEY_CHECKING=false; ansible -i inventory/qualys/qflow.ini kube-node --become --ask-pass -m file -a "path=/var/postgres-data state=directory"

Spark

kubectl create serviceaccount spark
kubectl create clusterrolebinding spark-role --clusterrole=edit --serviceaccount=default:spark --namespace=default