README.md

Apache Spark metrics extensions

This is a repository for ApacheSpark metrics related custom classes (e.g. sources, sinks). We were trying to extend the Spark Metrics subsystem with a Prometheus sink but the PR was not merged upstream. In order to support others to use Prometheus we have externalized the sink and made available through this repository, thus there is no need to build an Apache Spark fork.

• Prometheus sink

For further information how we use this extension and the Prometheus sink at Banzai Cloud please read these posts:

• Monitoring Apache Spark with Prometheus
  
• Monitoring multiple federated clusters with Prometheus - the secure way
  
• Application monitoring with Prometheus and Pipeline