Add a spark-shell smoke test to premerge and nightly [databricks]

[gerashegalov]

Contributes to #5704

This PR aims to catch issues like #9500. It modifies run_pyspark_from_build mostly to avoid recreating the logic of figuring out jar location etc.

Currently it may not catch this if do not have Spark 3.5.0 CI yet. But this is how it could reproduce the #9500

$ SPARK_HOME=~/dist/spark-3.1.1-bin-hadoop3.2 SPARK_SHELL_SMOKE_TEST=1 ./integration_tests/run_pyspark_from_build.sh
...
+ grep -F 'res0: Array[org.apache.spark.sql.Row] = Array([4950])'
res0: Array[org.apache.spark.sql.Row] = Array([4950])
+ echo 'SUCCESS spark-shell smoke test...'
SUCCESS spark-shell smoke test
$ echo $?
0

$ SPARK_HOME=~/dist/spark-3.5.0-bin-hadoop3 SPARK_SHELL_SMOKE_TEST=1 ./integration_tests/run_pyspark_from_build.sh
$ echo $?
1

SPARK_SHELL_SMOKE_TEST=1 \
  PYSP_TEST_spark_shuffle_manager=com.nvidia.spark.rapids.spark311.RapidsShuffleManager \
  SPARK_HOME=~/dist/spark-3.1.1-bin-hadoop3.2 \
  ./integration_tests/run_pyspark_from_build.sh
+ echo 'SUCCESS spark-shell smoke test'
SUCCESS spark-shell smoke test
$ echo $?
0

SPARK_SHELL_SMOKE_TEST=1 \
  PYSP_TEST_spark_shuffle_manager=com.nvidia.spark.rapids.spark350.RapidsShuffleManager \
  SPARK_HOME=~/dist/spark-3.5.0-bin-hadoop3 \
  ./integration_tests/run_pyspark_from_build.sh
$ echo $?
1

Signed-off-by: @gerashegalov

[gerashegalov]

build

[gerashegalov]

added shuffle manger conf to spark shell, PTAL

[gerashegalov]

build

[gerashegalov]

build

[gerashegalov]

build