-
Notifications
You must be signed in to change notification settings - Fork 14
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Added spark driver info to UI with alerts #15
base: main
Are you sure you want to change the base?
Added spark driver info to UI with alerts #15
Conversation
spark-plugin/plugin/src/main/scala/org/apache/spark/dataflint/listener/model.scala
Show resolved
Hide resolved
spark-plugin/plugin/src/main/scala/org/apache/spark/dataflint/DataflintSparkUILoader.scala
Outdated
Show resolved
Hide resolved
af13c7a
to
2c2ee4e
Compare
</Typography> | ||
<Typography variant="subtitle2"> | ||
"Memory Usage" is the peak memory usage of the most | ||
<Typography variant="body1" color="inherit" textAlign={"center"}> |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
when no data please hide the text
); | ||
const driverMaxMemory = environmentInfo?.driverXmxBytes ?? 1; | ||
const driverMemoryUsage = driverExecutor?.memoryUsageBytes ?? 0; | ||
const driverMemoryUsagePercentage = |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
please use the calculatePercentage util I wrote
else (listener: SparkListenerInterface, queue: String) => context.listenerBus.addToQueue(listener, queue) | ||
addToQueueMethod(dataflintListener, "dataflint") | ||
context.listenerBus.post(DataflintEnvironmentInfoEvent(environmentInfo)) | ||
logInfo(s"Posted DataflintEnvironmentInfoEvent max driver memory: ${driverXmxBytes}") |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
please remove this log line from production
logInfo(s"Posted DataflintEnvironmentInfoEvent max driver memory: ${driverXmxBytes}") | ||
if (isDatabricks) { | ||
addToQueueMethod(DataflintDatabricksLiveListener(context.listenerBus), "dataflint") | ||
logInfo("Added DataflintDatabricksLiveListener to the Spark context") |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
please remove this line from production
? Object.values(executors).find((exec) => exec.id === "driver") | ||
: undefined; | ||
const driverMaxMemory = environmentInfo?.driverXmxBytes ?? 1; | ||
const driverMemoryUsage = driverExecutor?.memoryUsageBytes ?? 0; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
please change memoryUsageBytes to use only on heap memory (+ executors)
Added spark driver info to the memory usage in the status and summary tabs
Added relevant alerts for driver over usage