Skip to content

Latest commit

 

History

History
141 lines (93 loc) · 6.59 KB

README.md

File metadata and controls

141 lines (93 loc) · 6.59 KB
Raw

Logo

Spark Performance Made Simple

Maven Package Slack Test Status Docs License

If you enjoy DataFlint please give us a ⭐️ and join our slack community for feature requests, support and more!

What is DataFlint?

DataFlint is a modern, user-friendly enhancement for Apache Spark that simplifies performance monitoring and debugging. It adds an intuitive tab to the existing Spark Web UI, transforming a powerful but often overwhelming interface into something easy to navigate and understand.

Why DataFlint?

  • Intuitive Design: DataFlint's tab in the Spark Web UI presents complex metrics in a clear, easy-to-understand format, making Spark performance accessible to everyone.
  • Effortless Setup: Install DataFlint in minutes with just a few lines of code or configuration, without making any changes to your existing Spark environment.
  • For All Skill Levels: Whether you're a seasoned data engineer or just starting with Spark, DataFlint provides valuable insights that help you work more effectively.

With DataFlint, spend less time deciphering Spark Web UI and more time deriving value from your data. Make big data work better for you, regardless of your role or experience level with Spark.

Usage

After installation, you will see a "DataFlint" tab in the Spark Web UI. Click on it to start using DataFlint.

Logo

Demo

Demo

Features

  • 📈 Real-time query and cluster status
  • 📊 Query breakdown with performance heat map
  • 📋 Application Run Summary
  • ⚠️ Performance alerts and suggestions
  • 👀 Identify query failures
  • 🤖 Spark AI Assistant

See Our Features for more information

Installation

Scala

Install DataFlint via sbt:

libraryDependencies += "io.dataflint" %% "spark" % "0.2.6"

Then instruct spark to load the DataFlint plugin:

val spark = SparkSession
    .builder()
    .config("spark.plugins", "io.dataflint.spark.SparkDataflintPlugin")
    ...
    .getOrCreate()

PySpark

Add these 2 configs to your pyspark session builder:

builder = pyspark.sql.SparkSession.builder
    ...
    .config("spark.jars.packages", "io.dataflint:spark_2.12:0.2.6") \
    .config("spark.plugins", "io.dataflint.spark.SparkDataflintPlugin") \
    ...

Spark Submit

Alternatively, install DataFlint with no code change as a spark ivy package by adding these 2 lines to your spark-submit command:

spark-submit
--packages io.dataflint:spark_2.12:0.2.6 \
--conf spark.plugins=io.dataflint.spark.SparkDataflintPlugin \
...

Additional installation options

  • There is also support for scala 2.13, if your spark cluster is using scala 2.13 change package name to io.dataflint:spark_2.13:0.2.6
  • For more installation options, including for python and k8s spark-operator, see Install on Spark docs
  • For installing DataFlint in spark history server for observability on completed runs see install on spark history server docs
  • For installing DataFlint on DataBricks see install on databricks docs

How it Works

How it Works

DataFlint is installed as a plugin on the spark driver and history server.

The plugin exposes an additional HTTP resoures for additional metrics not available in Spark UI, and a modern SPA web-app that fetches data from spark without the need to refresh the page.

For more information, see how it works docs

Medium Articles

Compatibility Matrix

DataFlint require spark version 3.2 and up, and supports both scala versions 2.12 or 2.13.

Spark Platforms DataFlint Realtime DataFlint History server
Local ✅ ✅
Standalone ✅ ✅
Kubernetes Spark Operator ✅ ✅
EMR ✅ ✅
Dataproc ✅ ❓
HDInsights ✅ ❓
Databricks ✅ ❌

For more information, see supported versions docs