.Net docs need to stop mentioning databricks

[dbeavon]

I opened a very unsatisfying support case with databricks (via azure-databricks). The end result is that they don't support spark-dotnet.

I spent a full day building a self-contained repro, and sent it thru azure support. The azure-databricks support team interacted extensively with project managers at databricks. However none of the "spark core" engineers were willing to spend time digging into the repro. As soon as they saw spark-dotnet in the stack frames they refused to look any deeper at the problem.

The challenge is that their "driver daemon" is proprietary and behaves differently than OSS spark. So if the engineers refuse to investigate, then there is no recourse. It is like running into a brick wall (pun intended).

I had given Terry a heads-up about this case, but I don't think anyone else contacted him. While I do believe that he has worked with databricks engineers in the past, I don't think they were ready to support any customers yet (especially indirect customers in Microsoft Azure).

I should soon have an authoritative statement about their lack of support for .Net.

@Niharikadutta @imback82

Is there any chance we can put warnings about supportability in these docs (links below)?
https://docs.microsoft.com/en-us/dotnet/spark/what-is-apache-spark-dotnet
https://docs.microsoft.com/en-us/dotnet/spark/tutorials/databricks-deployment

It seems to me that the docs are a bit misleading given they encourage the use of databricks, but the databricks team itself isn't willing to support .Net in any capacity whatsoever. Previously there were warnings on these pages (prior to v.1.0.0) but the warnings were all removed. That was the point when I thought it was safe to use databricks for .Net solutions. Given their lack of support I think it is critical to bring the warnings back again. Without support from the databricks engineers, there is no way to investigate why a solution that works on OSS spark will not work when deployed into their proprietary "driver daemon".

[imback82]

@dbeavon I had a call with Databricks engineers earlier this week. There are two things we are going to try:

1. From the supporting team, they will try to replicate the behavior with Scala/Python to see if this is related to Spark .NET or not.
2. I will add more logging to ThreadPool.scala and see what's causing the issue. I plan to share the changes with you by the end of this week so that you can try with the new JAR.

Previously there were warnings on these pages (prior to v.1.0.0) but the warnings were all removed.

@Niharikadutta / @luisquintanilla, can you answer to this quesion why the warning were removed?

[Niharikadutta]

Thanks @dbeavon for providing this statement. I will use this to update our documentation about the lack of support for .NET for Apache Spark on Databricks.

[Niharikadutta]

@imback82 please find the microsoft-spark jar with logging statements in ThreadPool.scala - microsoft-spark-3-0_2.12-1.1.1.jar.zip

[imback82]

Thanks @Niharikadutta! Can you share the branch with the changes as well?

@dbeavon could you try submitting your job with the JAR that @Niharikadutta shared above? This will help us debugging further.

[Niharikadutta]

Here are the changes - #936.

[dbeavon]

@imback82 @Niharikadutta Please give me another day or so.... I had to open another related support case with databricks. Their log4j stuff is crapping out on me. I believe that every "spark_jar_task", when initiated, is causing the log4j level to be elevated dynamically. This may be happening from some driver logic of theirs.

My log4j preferences for the cluster are being overridden. The logging config changes made by cluster init scripts , and my own dynamic changes to the logging level are both being overridden..

I don't know why I keep running into issues that should have been encountered by other scala developers! I'm guessing that the scala developers went somewhere else. Perhaps the only users on azure databricks may be "data scientists" and they are probably isolated to using their python notebooks. They may have lower expectations, and/or may not even care about their log4j logging.