issue in saving the dataframe to a local file

[visguru1970]

I have a small code
static void Main(string[] args)
{
var spark = SparkSession
.Builder()
.AppName("DemoApp")
.GetOrCreate();
var dataFrame = spark.Sql("select id, round(rand()*100,2) as random_number from range(10)");
dataFrame.Show();

        dataFrame.Coalesce(1)
            .Write()
            .Format("csv")
            .Option("header", true)
            .Option("sep", ",")
            .Mode("overwrite")
            .Save(@"C:\mytest");
        spark.Stop();
    }
  It creates the folder called mytest but not writing any thing. I am using Spark 3.1.2 and getting error

n. System.Exception: JVM method execution failed: Nonstatic method 'save' failed for class '9' when called with 1 arguments ([Index=1, Type=String, Value=C:\mytest],

[visguru1970]

This is using Spark with .NET 3.1

[imback82]

@visguru1970 Can you share the full log?

[visguru1970]

I have downloaded the following spark version spark-3.1.2-bin-hadoop3.2.rar from the sparksite. so the spark version is 3.1.2. My java version is java version "1.8.0_301".
I have extracted the files from spark-3.1.2-bin-hadoop3.2.rar and copied them to a folder BigDataLocalSetup/spark folder and also copied the winutils.exe latest to the bin folder which is under spark. so these are my environment variables

JAVA_HOME C:\Program Files\Java\jdk1.8.0_301
SPARK_HOME C:\BigDataLocalSetup\spark
HADOOP_HOME C:\BigDataLocalSetup\spark

When i try spark-submit --version i am getting Spark 3.1.2 version

when i installed the The newget libraries for the latest apache spark, these are the following jars got installed
microsoft-spark-2-4_2.11-2.0.0.jar
microsoft-spark-3-0_2.12-2.0.0.jar
microsoft-spark-3-1_2.12-2.0.0.jar

When i debugged using the following statement
spark-submit --class org.apache.spark.deploy.dotnet.DotnetRunner --master local bin\Debug\netcoreapp3.1\microsoft-spark-3-1_2.12-2.0.0.jar debug

Inner exception

•   $exception	InnerException = {"org.apache.spark.SparkException: Job aborted.\r\n\tat org.apache.spark.sql.execution.datasources.FileFormatWriter$.write(FileFormatWriter.scala:231)\r\n\tat org.apache.spark.sql.execution.datasources.InsertIntoHadoopFsRelationCommand.run(InsertIntoHadoopFsRelationCommand.scala:188)\r\n\tat org.apache.spark.sql.execution.command.DataWritingCommandExec.sideEffectResult$lzycompute(commands.scala:108)\r\n\tat org.apache.spark.sql.execution.command.DataWritingCommandExec.sideEffectResult(commands.scala:106)\r\n\tat org.apache.spark.sql.execution.command.DataWritingCommandExec.doExecute(commands.scala:131)\r\n\tat org.apache.spark.sql.execution.SparkPlan.$anonfun$execute$1(SparkPlan.scala:180)\r\n\tat org.apache.spark.sql.execution.SparkPlan.$anonfun$executeQuery$1(SparkPlan.scala:218)\r\n\tat org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)\r\n\tat org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:215)\r\n\tat org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:176)\r\n\tat org.apache.spark.sql.execution.QueryExecution.toRdd$lzycompute(QueryExecution.scala:132)\r\n\tat org.apache.spark.sql.execution.QueryExecution.toRdd(QueryExecution.scala:131)\r\n\tat org.apache.spark.sql.DataFrameWriter.$anonfun$runCommand$1(DataFrameWriter.scala:989)\r\n\tat org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$5(SQLExecution.scala:103)\r\n\tat org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:163)\r\n\tat org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$1(SQLExecution.scala:90)\r\n\tat org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:775)\r\n\tat org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:64)\r\n\tat org.apache.spark.sql.DataFrameWriter.runCommand(DataFrameWriter.scala:989)\r\n\tat org.apache.spark.sql.DataFrameWriter.saveToV1Source(DataFrameWriter.scala:438)\r\n\tat org.apache.spark.sql.DataFrameWriter.saveInternal(DataFrameWriter.scala:415)\r\n\tat org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:293)\r\n\tat sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)\r\n\tat sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)\r\n\tat sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\r\n\tat java.lang.reflect.Method.invoke(Method.java:498)\r\n\tat org.apache.spark.api.dotnet.DotnetBackendHandler.handleMethodCall(DotnetBackendHandler.scala:165)\r\n\tat org.apache.spark.api.dotnet.DotnetBackendHandler.$anonfun$handleBackendRequest$2(DotnetBackendHandler.scala:105)\r\n\tat org.apache.spark.api.dotnet.ThreadPool$$anon$1.run(ThreadPool.scala:34)\r\n\tat java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)\r\n\tat java.util.concurrent.FutureTask.run(FutureTask.java:266)\r\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)\r\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)\r\n\tat java.lang.Thread.run(Thread.java:748)\r\nCaused by: java.lang.UnsatisfiedLinkError: org.apache.hadoop.io.nativeio.NativeIO$Windows.access0(Ljava/lang/String;I)Z\r\n\tat org.apache.hadoop.io.nativeio.NativeIO$Windows.access0(Native Method)\r\n\tat org.apache.hadoop.io.nativeio.NativeIO$Windows.access(NativeIO.java:645)\r\n\tat org.apache.hadoop.fs.FileUtil.canRead(FileUtil.java:1230)\r\n\tat org.apache.hadoop.fs.FileUtil.list(FileUtil.java:1435)\r\n\tat org.apache.hadoop.fs.RawLocalFileSystem.listStatus(RawLocalFileSystem.java:493)\r\n\tat org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:1868)\r\n\tat org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:1910)\r\n\tat org.apache.hadoop.fs.ChecksumFileSystem.listStatus(ChecksumFileSystem.java:678)\r\n\tat org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:1868)\r\n\tat org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:1910)\r\n\tat org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.getAllCommittedTaskPaths(FileOutputCommitter.java:332)\r\n\tat org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.commitJobInternal(FileOutputCommitter.java:402)\r\n\tat org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.commitJob(FileOutputCommitter.java:375)\r\n\tat org.apache.spark.internal.io.HadoopMapReduceCommitProtocol.commitJob(HadoopMapReduceCommitProtocol.scala:182)\r\n\tat org.apache.spark.sql.execution.datasources.FileFormatWriter$.write(FileFormatWriter.scala:220)\r\n\t... 33 more\r\n"}	System.Exception
  

[imback82]

Can you attach a full log (not just exception)?

[visguru1970]

C:\CaliforniaHousing>spark-submit --class org.apache.spark.deploy.dotnet.DotnetRunner --master local bin\Debug\netcoreapp3.1\microsoft-spark-3-1_2.12-2.0.0.jar dotnet bin\Debug\netcoreapp3.1\CaliforniaHousing.dll 21/10/24 19:56:03 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties 21/10/24 19:56:03 INFO DotnetRunner: Starting DotnetBackend with dotnet. 21/10/24 19:56:03 INFO DotnetBackend: The number of DotnetBackend threads is set to 10. 21/10/24 19:56:04 INFO DotnetRunner: Port number used by DotnetBackend is 54383 21/10/24 19:56:04 INFO DotnetRunner: Adding key=spark.jars and value=file:/C:/CaliforniaHousing/bin/Debug/netcoreapp3.1/microsoft-spark-3-1_2.12-2.0.0.jar to environment 21/10/24 19:56:04 INFO DotnetRunner: Adding key=spark.app.name and value=org.apache.spark.deploy.dotnet.DotnetRunner to environment 21/10/24 19:56:04 INFO DotnetRunner: Adding key=spark.submit.pyFiles and value= to environment 21/10/24 19:56:04 INFO DotnetRunner: Adding key=spark.submit.deployMode and value=client to environment 21/10/24 19:56:04 INFO DotnetRunner: Adding key=spark.master and value=local to environment [2021-10-24T23:56:05.2758690Z] [VISHPC] [Info] [ConfigurationService] Using port 54383 for connection. [2021-10-24T23:56:05.2862762Z] [VISHPC] [Info] [JvmBridge] JvMBridge port is 54383 [2021-10-24T23:56:05.2898540Z] [VISHPC] [Info] [JvmBridge] The number of JVM backend thread is set to 10. The max number of concurrent sockets in JvmBridge is set to 7. 21/10/24 19:56:05 INFO SparkContext: Running Spark version 3.1.2 21/10/24 19:56:05 INFO ResourceUtils: ============================================================== 21/10/24 19:56:05 INFO ResourceUtils: No custom resources configured for spark.driver. 21/10/24 19:56:05 INFO ResourceUtils: ============================================================== 21/10/24 19:56:05 INFO SparkContext: Submitted application: DemoApp 21/10/24 19:56:05 INFO ResourceProfile: Default ResourceProfile created, executor resources: Map(cores -> name: cores, amount: 1, script: , vendor: , memory -> name: memory, amount: 1024, script: , vendor: , offHeap -> name: offHeap, amount: 0, script: , vendor: ), task resources: Map(cpus -> name: cpus, amount: 1.0) 21/10/24 19:56:05 INFO ResourceProfile: Limiting resource is cpu 21/10/24 19:56:05 INFO ResourceProfileManager: Added ResourceProfile id: 0 21/10/24 19:56:05 INFO SecurityManager: Changing view acls to: Dell 21/10/24 19:56:05 INFO SecurityManager: Changing modify acls to: Dell 21/10/24 19:56:05 INFO SecurityManager: Changing view acls groups to: 21/10/24 19:56:05 INFO SecurityManager: Changing modify acls groups to: 21/10/24 19:56:05 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(Dell); groups with view permissions: Set(); users with modify permissions: Set(Dell); groups with modify permissions: Set() 21/10/24 19:56:05 INFO Utils: Successfully started service 'sparkDriver' on port 54391. 21/10/24 19:56:05 INFO SparkEnv: Registering MapOutputTracker 21/10/24 19:56:06 INFO SparkEnv: Registering BlockManagerMaster 21/10/24 19:56:06 INFO BlockManagerMasterEndpoint: Using org.apache.spark.storage.DefaultTopologyMapper for getting topology information 21/10/24 19:56:06 INFO BlockManagerMasterEndpoint: BlockManagerMasterEndpoint up 21/10/24 19:56:06 INFO SparkEnv: Registering BlockManagerMasterHeartbeat 21/10/24 19:56:06 INFO DiskBlockManager: Created local directory at C:\Users\Dell\AppData\Local\Temp\blockmgr-9c117f7b-75a5-4688-a98e-9dee2af5129b 21/10/24 19:56:06 INFO MemoryStore: MemoryStore started with capacity 366.3 MiB 21/10/24 19:56:06 INFO SparkEnv: Registering OutputCommitCoordinator 21/10/24 19:56:06 INFO Utils: Successfully started service 'SparkUI' on port 4040. 21/10/24 19:56:06 INFO SparkUI: Bound SparkUI to 0.0.0.0, and started at http://localhost:4040 21/10/24 19:56:06 INFO SparkContext: Added JAR file:/C:/CaliforniaHousing/bin/Debug/netcoreapp3.1/microsoft-spark-3-1_2.12-2.0.0.jar at spark://localhost:54391/jars/microsoft-spark-3-1_2.12-2.0.0.jar with timestamp 1635119765584 21/10/24 19:56:07 INFO Executor: Starting executor ID driver on host localhost 21/10/24 19:56:07 INFO Executor: Fetching spark://localhost:54391/jars/microsoft-spark-3-1_2.12-2.0.0.jar with timestamp 1635119765584 21/10/24 19:56:07 INFO TransportClientFactory: Successfully created connection to localhost/127.0.0.1:54391 after 38 ms (0 ms spent in bootstraps) 21/10/24 19:56:07 INFO Utils: Fetching spark://localhost:54391/jars/microsoft-spark-3-1_2.12-2.0.0.jar to C:\Users\Dell\AppData\Local\Temp\spark-48a006a1-5598-4b8e-8309-dcbbaa6c3914\userFiles-9fb96294-452c-4191-9d5f-162c43c4fecf\fetchFileTemp1897026340048289274.tmp 21/10/24 19:56:07 INFO Executor: Adding file:/C:/Users/Dell/AppData/Local/Temp/spark-48a006a1-5598-4b8e-8309-dcbbaa6c3914/userFiles-9fb96294-452c-4191-9d5f-162c43c4fecf/microsoft-spark-3-1_2.12-2.0.0.jar to class loader 21/10/24 19:56:07 INFO Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 54407. 21/10/24 19:56:07 INFO NettyBlockTransferService: Server created on localhost:54407 21/10/24 19:56:07 INFO BlockManager: Using org.apache.spark.storage.RandomBlockReplicationPolicy for block replication policy 21/10/24 19:56:07 INFO BlockManagerMaster: Registering BlockManager BlockManagerId(driver, localhost, 54407, None) 21/10/24 19:56:07 INFO BlockManagerMasterEndpoint: Registering block manager localhost:54407 with 366.3 MiB RAM, BlockManagerId(driver, localhost, 54407, None) 21/10/24 19:56:07 INFO BlockManagerMaster: Registered BlockManager BlockManagerId(driver, localhost, 54407, None) 21/10/24 19:56:07 INFO BlockManager: Initialized BlockManager: BlockManagerId(driver, localhost, 54407, None) 21/10/24 19:56:08 INFO SharedState: Setting hive.metastore.warehouse.dir ('null') to the value of spark.sql.warehouse.dir ('file:/C:/CaliforniaHousing/spark-warehouse'). 21/10/24 19:56:08 INFO SharedState: Warehouse path is 'file:/C:/CaliforniaHousing/spark-warehouse'. 21/10/24 19:56:14 INFO CodeGenerator: Code generated in 420.9075 ms 21/10/24 19:56:14 INFO SparkContext: Starting job: showString at NativeMethodAccessorImpl.java:0 21/10/24 19:56:14 INFO DAGScheduler: Got job 0 (showString at NativeMethodAccessorImpl.java:0) with 1 output partitions 21/10/24 19:56:14 INFO DAGScheduler: Final stage: ResultStage 0 (showString at NativeMethodAccessorImpl.java:0) 21/10/24 19:56:14 INFO DAGScheduler: Parents of final stage: List() 21/10/24 19:56:14 INFO DAGScheduler: Missing parents: List() 21/10/24 19:56:14 INFO DAGScheduler: Submitting ResultStage 0 (MapPartitionsRDD[3] at showString at NativeMethodAccessorImpl.java:0), which has no missing parents 21/10/24 19:56:15 INFO MemoryStore: Block broadcast_0 stored as values in memory (estimated size 11.1 KiB, free 366.3 MiB) 21/10/24 19:56:15 INFO MemoryStore: Block broadcast_0_piece0 stored as bytes in memory (estimated size 5.1 KiB, free 366.3 MiB) 21/10/24 19:56:15 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on localhost:54407 (size: 5.1 KiB, free: 366.3 MiB) 21/10/24 19:56:15 INFO SparkContext: Created broadcast 0 from broadcast at DAGScheduler.scala:1388 21/10/24 19:56:15 INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 0 (MapPartitionsRDD[3] at showString at NativeMethodAccessorImpl.java:0) (first 15 tasks are for partitions Vector(0)) 21/10/24 19:56:15 INFO TaskSchedulerImpl: Adding task set 0.0 with 1 tasks resource profile 0 21/10/24 19:56:15 INFO TaskSetManager: Starting task 0.0 in stage 0.0 (TID 0) (localhost, executor driver, partition 0, PROCESS_LOCAL, 4578 bytes) taskResourceAssignments Map() 21/10/24 19:56:15 INFO Executor: Running task 0.0 in stage 0.0 (TID 0) 21/10/24 19:56:15 INFO Executor: Finished task 0.0 in stage 0.0 (TID 0). 1561 bytes result sent to driver 21/10/24 19:56:15 INFO TaskSetManager: Finished task 0.0 in stage 0.0 (TID 0) in 578 ms on localhost (executor driver) (1/1) 21/10/24 19:56:15 INFO TaskSchedulerImpl: Removed TaskSet 0.0, whose tasks have all completed, from pool 21/10/24 19:56:15 INFO DAGScheduler: ResultStage 0 (showString at NativeMethodAccessorImpl.java:0) finished in 0.991 s 21/10/24 19:56:15 INFO DAGScheduler: Job 0 is finished. Cancelling potential speculative or zombie tasks for this job 21/10/24 19:56:15 INFO TaskSchedulerImpl: Killing all running tasks in stage 0: Stage finished 21/10/24 19:56:15 INFO DAGScheduler: Job 0 finished: showString at NativeMethodAccessorImpl.java:0, took 1.090930 s 21/10/24 19:56:16 INFO CodeGenerator: Code generated in 35.5192 ms 21/10/24 19:56:16 INFO BlockManagerInfo: Removed broadcast_0_piece0 on localhost:54407 in memory (size: 5.1 KiB, free: 366.3 MiB) +---+-------------+ | id|random_number| +---+-------------+ | 0| 56.79| | 1| 43.33| | 2| 53.94| | 3| 90.07| | 4| 63.79| | 5| 84.94| | 6| 56.91| | 7| 45.13| | 8| 6.61| | 9| 63.09| +---+-------------+ 21/10/24 19:56:16 INFO FileOutputCommitter: File Output Committer Algorithm version is 1 21/10/24 19:56:16 INFO FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false 21/10/24 19:56:16 INFO SQLHadoopMapReduceCommitProtocol: Using output committer class org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter 21/10/24 19:56:17 INFO CodeGenerator: Code generated in 73.0345 ms 21/10/24 19:56:17 INFO SparkContext: Starting job: save at NativeMethodAccessorImpl.java:0 21/10/24 19:56:17 INFO DAGScheduler: Got job 1 (save at NativeMethodAccessorImpl.java:0) with 1 output partitions 21/10/24 19:56:17 INFO DAGScheduler: Final stage: ResultStage 1 (save at NativeMethodAccessorImpl.java:0) 21/10/24 19:56:17 INFO DAGScheduler: Parents of final stage: List() 21/10/24 19:56:17 INFO DAGScheduler: Missing parents: List() 21/10/24 19:56:17 INFO DAGScheduler: Submitting ResultStage 1 (CoalescedRDD[10] at save at NativeMethodAccessorImpl.java:0), which has no missing parents 21/10/24 19:56:17 INFO MemoryStore: Block broadcast_1 stored as values in memory (estimated size 175.2 KiB, free 366.1 MiB) 21/10/24 19:56:17 INFO MemoryStore: Block broadcast_1_piece0 stored as bytes in memory (estimated size 62.7 KiB, free 366.1 MiB) 21/10/24 19:56:17 INFO BlockManagerInfo: Added broadcast_1_piece0 in memory on localhost:54407 (size: 62.7 KiB, free: 366.2 MiB) 21/10/24 19:56:17 INFO SparkContext: Created broadcast 1 from broadcast at DAGScheduler.scala:1388 21/10/24 19:56:17 INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 1 (CoalescedRDD[10] at save at NativeMethodAccessorImpl.java:0) (first 15 tasks are for partitions Vector(0)) 21/10/24 19:56:17 INFO TaskSchedulerImpl: Adding task set 1.0 with 1 tasks resource profile 0 21/10/24 19:56:17 INFO TaskSetManager: Starting task 0.0 in stage 1.0 (TID 1) (localhost, executor driver, partition 0, PROCESS_LOCAL, 4834 bytes) taskResourceAssignments Map() 21/10/24 19:56:17 INFO Executor: Running task 0.0 in stage 1.0 (TID 1) 21/10/24 19:56:17 INFO FileOutputCommitter: File Output Committer Algorithm version is 1 21/10/24 19:56:17 INFO FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false 21/10/24 19:56:17 INFO SQLHadoopMapReduceCommitProtocol: Using output committer class org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter 21/10/24 19:56:17 INFO FileOutputCommitter: Saved output of task 'attempt_202110241956171300391628461070745_0001_m_000000_1' to file:/C:/mytest/_temporary/0/task_202110241956171300391628461070745_0001_m_000000 21/10/24 19:56:17 INFO SparkHadoopMapRedUtil: attempt_202110241956171300391628461070745_0001_m_000000_1: Committed 21/10/24 19:56:17 INFO Executor: Finished task 0.0 in stage 1.0 (TID 1). 2439 bytes result sent to driver 21/10/24 19:56:17 INFO TaskSetManager: Finished task 0.0 in stage 1.0 (TID 1) in 493 ms on localhost (executor driver) (1/1) 21/10/24 19:56:17 INFO DAGScheduler: ResultStage 1 (save at NativeMethodAccessorImpl.java:0) finished in 0.561 s 21/10/24 19:56:17 INFO TaskSchedulerImpl: Removed TaskSet 1.0, whose tasks have all completed, from pool 21/10/24 19:56:17 INFO DAGScheduler: Job 1 is finished. Cancelling potential speculative or zombie tasks for this job 21/10/24 19:56:17 INFO TaskSchedulerImpl: Killing all running tasks in stage 1: Stage finished 21/10/24 19:56:17 INFO DAGScheduler: Job 1 finished: save at NativeMethodAccessorImpl.java:0, took 0.578143 s 21/10/24 19:56:17 ERROR FileFormatWriter: Aborting job 99d4e620-0037-45e8-8434-2f405a7db34f. java.lang.UnsatisfiedLinkError: org.apache.hadoop.io.nativeio.NativeIO$Windows.access0(Ljava/lang/String;I)Z at org.apache.hadoop.io.nativeio.NativeIO$Windows.access0(Native Method) at org.apache.hadoop.io.nativeio.NativeIO$Windows.access(NativeIO.java:645) at org.apache.hadoop.fs.FileUtil.canRead(FileUtil.java:1230) at org.apache.hadoop.fs.FileUtil.list(FileUtil.java:1435) at org.apache.hadoop.fs.RawLocalFileSystem.listStatus(RawLocalFileSystem.java:493) at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:1868) at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:1910) at org.apache.hadoop.fs.ChecksumFileSystem.listStatus(ChecksumFileSystem.java:678) at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:1868) at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:1910) at org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.getAllCommittedTaskPaths(FileOutputCommitter.java:332) at org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.commitJobInternal(FileOutputCommitter.java:402) at org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.commitJob(FileOutputCommitter.java:375) at org.apache.spark.internal.io.HadoopMapReduceCommitProtocol.commitJob(HadoopMapReduceCommitProtocol.scala:182) at org.apache.spark.sql.execution.datasources.FileFormatWriter$.write(FileFormatWriter.scala:220) at org.apache.spark.sql.execution.datasources.InsertIntoHadoopFsRelationCommand.run(InsertIntoHadoopFsRelationCommand.scala:188) at org.apache.spark.sql.execution.command.DataWritingCommandExec.sideEffectResult$lzycompute(commands.scala:108) at org.apache.spark.sql.execution.command.DataWritingCommandExec.sideEffectResult(commands.scala:106) at org.apache.spark.sql.execution.command.DataWritingCommandExec.doExecute(commands.scala:131) at org.apache.spark.sql.execution.SparkPlan.$anonfun$execute$1(SparkPlan.scala:180) at org.apache.spark.sql.execution.SparkPlan.$anonfun$executeQuery$1(SparkPlan.scala:218) at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151) at org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:215) at org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:176) at org.apache.spark.sql.execution.QueryExecution.toRdd$lzycompute(QueryExecution.scala:132) at org.apache.spark.sql.execution.QueryExecution.toRdd(QueryExecution.scala:131) at org.apache.spark.sql.DataFrameWriter.$anonfun$runCommand$1(DataFrameWriter.scala:989) at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$5(SQLExecution.scala:103) at org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:163) at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$1(SQLExecution.scala:90) at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:775) at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:64) at org.apache.spark.sql.DataFrameWriter.runCommand(DataFrameWriter.scala:989) at org.apache.spark.sql.DataFrameWriter.saveToV1Source(DataFrameWriter.scala:438) at org.apache.spark.sql.DataFrameWriter.saveInternal(DataFrameWriter.scala:415) at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:293) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.spark.api.dotnet.DotnetBackendHandler.handleMethodCall(DotnetBackendHandler.scala:165) at org.apache.spark.api.dotnet.DotnetBackendHandler.$anonfun$handleBackendRequest$2(DotnetBackendHandler.scala:105) at org.apache.spark.api.dotnet.ThreadPool$$anon$1.run(ThreadPool.scala:34) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 21/10/24 19:56:17 ERROR DotnetBackendHandler: Failed to execute 'save' on 'org.apache.spark.sql.DataFrameWriter' with args=([Type=java.lang.String, Value: C:\mytest]) [2021-10-24T23:56:17.9409253Z] [VISHPC] [Error] [JvmBridge] JVM method execution failed: Nonstatic method 'save' failed for class '9' when called with 1 arguments ([Index=1, Type=String, Value=C:\mytest], ) [2021-10-24T23:56:17.9411093Z] [VISHPC] [Error] [JvmBridge] org.apache.spark.SparkException: Job aborted. at org.apache.spark.sql.execution.datasources.FileFormatWriter$.write(FileFormatWriter.scala:231) at org.apache.spark.sql.execution.datasources.InsertIntoHadoopFsRelationCommand.run(InsertIntoHadoopFsRelationCommand.scala:188) at org.apache.spark.sql.execution.command.DataWritingCommandExec.sideEffectResult$lzycompute(commands.scala:108) at org.apache.spark.sql.execution.command.DataWritingCommandExec.sideEffectResult(commands.scala:106) at org.apache.spark.sql.execution.command.DataWritingCommandExec.doExecute(commands.scala:131) at org.apache.spark.sql.execution.SparkPlan.$anonfun$execute$1(SparkPlan.scala:180) at org.apache.spark.sql.execution.SparkPlan.$anonfun$executeQuery$1(SparkPlan.scala:218) at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151) at org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:215) at org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:176) at org.apache.spark.sql.execution.QueryExecution.toRdd$lzycompute(QueryExecution.scala:132) at org.apache.spark.sql.execution.QueryExecution.toRdd(QueryExecution.scala:131) at org.apache.spark.sql.DataFrameWriter.$anonfun$runCommand$1(DataFrameWriter.scala:989) at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$5(SQLExecution.scala:103) at org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:163) at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$1(SQLExecution.scala:90) at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:775) at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:64) at org.apache.spark.sql.DataFrameWriter.runCommand(DataFrameWriter.scala:989) at org.apache.spark.sql.DataFrameWriter.saveToV1Source(DataFrameWriter.scala:438) at org.apache.spark.sql.DataFrameWriter.saveInternal(DataFrameWriter.scala:415) at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:293) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.spark.api.dotnet.DotnetBackendHandler.handleMethodCall(DotnetBackendHandler.scala:165) at org.apache.spark.api.dotnet.DotnetBackendHandler.$anonfun$handleBackendRequest$2(DotnetBackendHandler.scala:105) at org.apache.spark.api.dotnet.ThreadPool$$anon$1.run(ThreadPool.scala:34) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: java.lang.UnsatisfiedLinkError: org.apache.hadoop.io.nativeio.NativeIO$Windows.access0(Ljava/lang/String;I)Z at org.apache.hadoop.io.nativeio.NativeIO$Windows.access0(Native Method) at org.apache.hadoop.io.nativeio.NativeIO$Windows.access(NativeIO.java:645) at org.apache.hadoop.fs.FileUtil.canRead(FileUtil.java:1230) at org.apache.hadoop.fs.FileUtil.list(FileUtil.java:1435) at org.apache.hadoop.fs.RawLocalFileSystem.listStatus(RawLocalFileSystem.java:493) at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:1868) at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:1910) at org.apache.hadoop.fs.ChecksumFileSystem.listStatus(ChecksumFileSystem.java:678) at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:1868) at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:1910) at org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.getAllCommittedTaskPaths(FileOutputCommitter.java:332) at org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.commitJobInternal(FileOutputCommitter.java:402) at org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.commitJob(FileOutputCommitter.java:375) at org.apache.spark.internal.io.HadoopMapReduceCommitProtocol.commitJob(HadoopMapReduceCommitProtocol.scala:182) at org.apache.spark.sql.execution.datasources.FileFormatWriter$.write(FileFormatWriter.scala:220) ... 33 more [2021-10-24T23:56:17.9813024Z] [VISHPC] [Exception] [JvmBridge] JVM method execution failed: Nonstatic method 'save' failed for class '9' when called with 1 arguments ([Index=1, Type=String, Value=C:\mytest], ) at Microsoft.Spark.Interop.Ipc.JvmBridge.CallJavaMethod(Boolean isStatic, Object classNameOrJvmObjectReference, String methodName, Object[] args) Unhandled exception. System.Exception: JVM method execution failed: Nonstatic method 'save' failed for class '9' when called with 1 arguments ([Index=1, Type=String, Value=C:\mytest], ) ---> Microsoft.Spark.JvmException: org.apache.spark.SparkException: Job aborted. at org.apache.spark.sql.execution.datasources.FileFormatWriter$.write(FileFormatWriter.scala:231) at org.apache.spark.sql.execution.datasources.InsertIntoHadoopFsRelationCommand.run(InsertIntoHadoopFsRelationCommand.scala:188) at org.apache.spark.sql.execution.command.DataWritingCommandExec.sideEffectResult$lzycompute(commands.scala:108) at org.apache.spark.sql.execution.command.DataWritingCommandExec.sideEffectResult(commands.scala:106) at org.apache.spark.sql.execution.command.DataWritingCommandExec.doExecute(commands.scala:131) at org.apache.spark.sql.execution.SparkPlan.$anonfun$execute$1(SparkPlan.scala:180) at org.apache.spark.sql.execution.SparkPlan.$anonfun$executeQuery$1(SparkPlan.scala:218) at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151) at org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:215) at org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:176) at org.apache.spark.sql.execution.QueryExecution.toRdd$lzycompute(QueryExecution.scala:132) at org.apache.spark.sql.execution.QueryExecution.toRdd(QueryExecution.scala:131) at org.apache.spark.sql.DataFrameWriter.$anonfun$runCommand$1(DataFrameWriter.scala:989) at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$5(SQLExecution.scala:103) at org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:163) at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$1(SQLExecution.scala:90) at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:775) at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:64) at org.apache.spark.sql.DataFrameWriter.runCommand(DataFrameWriter.scala:989) at org.apache.spark.sql.DataFrameWriter.saveToV1Source(DataFrameWriter.scala:438) at org.apache.spark.sql.DataFrameWriter.saveInternal(DataFrameWriter.scala:415) at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:293) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.spark.api.dotnet.DotnetBackendHandler.handleMethodCall(DotnetBackendHandler.scala:165) at org.apache.spark.api.dotnet.DotnetBackendHandler.$anonfun$handleBackendRequest$2(DotnetBackendHandler.scala:105) at org.apache.spark.api.dotnet.ThreadPool$$anon$1.run(ThreadPool.scala:34) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: java.lang.UnsatisfiedLinkError: org.apache.hadoop.io.nativeio.NativeIO$Windows.access0(Ljava/lang/String;I)Z at org.apache.hadoop.io.nativeio.NativeIO$Windows.access0(Native Method) at org.apache.hadoop.io.nativeio.NativeIO$Windows.access(NativeIO.java:645) at org.apache.hadoop.fs.FileUtil.canRead(FileUtil.java:1230) at org.apache.hadoop.fs.FileUtil.list(FileUtil.java:1435) at org.apache.hadoop.fs.RawLocalFileSystem.listStatus(RawLocalFileSystem.java:493) at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:1868) at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:1910) at org.apache.hadoop.fs.ChecksumFileSystem.listStatus(ChecksumFileSystem.java:678) at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:1868) at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:1910) at org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.getAllCommittedTaskPaths(FileOutputCommitter.java:332) at org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.commitJobInternal(FileOutputCommitter.java:402) at org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.commitJob(FileOutputCommitter.java:375) at org.apache.spark.internal.io.HadoopMapReduceCommitProtocol.commitJob(HadoopMapReduceCommitProtocol.scala:182) at org.apache.spark.sql.execution.datasources.FileFormatWriter$.write(FileFormatWriter.scala:220) ... 33 more

…

--- End of inner exception stack trace --- at Microsoft.Spark.Interop.Ipc.JvmBridge.CallJavaMethod(Boolean isStatic, Object classNameOrJvmObjectReference, String methodName, Object[] args) at Microsoft.Spark.Interop.Ipc.JvmBridge.CallJavaMethod(Boolean isStatic, Object classNameOrJvmObjectReference, String methodName, Object arg0) at Microsoft.Spark.Interop.Ipc.JvmBridge.CallNonStaticJavaMethod(JvmObjectReference jvmObject, String methodName, Object arg0) at Microsoft.Spark.Interop.Ipc.JvmObjectReference.Invoke(String methodName, Object arg0) at Microsoft.Spark.Sql.DataFrameWriter.Save(String path) at CaliforniaHousing.Program.Main(String[] args) in C:\CaliforniaHousing\Program.cs:line 21 21/10/24 19:56:19 WARN ProcfsMetricsGetter: Exception when trying to compute pagesize, as a result reporting of ProcessTree metrics is stopped 21/10/24 19:56:19 INFO DotnetRunner: Closing DotnetBackend 21/10/24 19:56:19 INFO DotnetBackend: Callback server has already been shutdown. 21/10/24 19:56:19 INFO SparkUI: Stopped Spark web UI at http://localhost:4040 21/10/24 19:56:20 INFO MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped! 21/10/24 19:56:20 INFO MemoryStore: MemoryStore cleared 21/10/24 19:56:20 INFO BlockManager: BlockManager stopped 21/10/24 19:56:20 INFO BlockManagerMaster: BlockManagerMaster stopped 21/10/24 19:56:20 INFO OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped! 21/10/24 19:56:20 WARN SparkEnv: Exception while deleting Spark temp dir: C:\Users\Dell\AppData\Local\Temp\spark-48a006a1-5598-4b8e-8309-dcbbaa6c3914\userFiles-9fb96294-452c-4191-9d5f-162c43c4fecf java.io.IOException: Failed to delete: C:\Users\Dell\AppData\Local\Temp\spark-48a006a1-5598-4b8e-8309-dcbbaa6c3914\userFiles-9fb96294-452c-4191-9d5f-162c43c4fecf\microsoft-spark-3-1_2.12-2.0.0.jar at org.apache.spark.network.util.JavaUtils.deleteRecursivelyUsingJavaIO(JavaUtils.java:144) at org.apache.spark.network.util.JavaUtils.deleteRecursively(JavaUtils.java:118) at org.apache.spark.network.util.JavaUtils.deleteRecursivelyUsingJavaIO(JavaUtils.java:128) at org.apache.spark.network.util.JavaUtils.deleteRecursively(JavaUtils.java:118) at org.apache.spark.network.util.JavaUtils.deleteRecursively(JavaUtils.java:91) at org.apache.spark.util.Utils$.deleteRecursively(Utils.scala:1141) at org.apache.spark.SparkEnv.stop(SparkEnv.scala:105) at org.apache.spark.SparkContext.$anonfun$stop$23(SparkContext.scala:2108) at org.apache.spark.util.Utils$.tryLogNonFatalError(Utils.scala:1419) at org.apache.spark.SparkContext.stop(SparkContext.scala:2108) at org.apache.spark.deploy.SparkSubmit.$anonfun$runMain$13(SparkSubmit.scala:959) at org.apache.spark.deploy.SparkSubmit.$anonfun$runMain$13$adapted(SparkSubmit.scala:959) at scala.Option.foreach(Option.scala:407) at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:959) at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:180) at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:203) at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:90) at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1039) at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1048) at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) 21/10/24 19:56:20 INFO SparkContext: Successfully stopped SparkContext 21/10/24 19:56:20 INFO ShutdownHookManager: Shutdown hook called 21/10/24 19:56:20 INFO ShutdownHookManager: Deleting directory C:\Users\Dell\AppData\Local\Temp\spark-6758c994-f513-4ca8-b54d-60c18ef2581c 21/10/24 19:56:20 INFO ShutdownHookManager: Deleting directory C:\Users\Dell\AppData\Local\Temp\spark-48a006a1-5598-4b8e-8309-dcbbaa6c3914 21/10/24 19:56:20 ERROR ShutdownHookManager: Exception while deleting Spark temp dir: C:\Users\Dell\AppData\Local\Temp\spark-48a006a1-5598-4b8e-8309-dcbbaa6c3914 java.io.IOException: Failed to delete: C:\Users\Dell\AppData\Local\Temp\spark-48a006a1-5598-4b8e-8309-dcbbaa6c3914\userFiles-9fb96294-452c-4191-9d5f-162c43c4fecf\microsoft-spark-3-1_2.12-2.0.0.jar at org.apache.spark.network.util.JavaUtils.deleteRecursivelyUsingJavaIO(JavaUtils.java:144) at org.apache.spark.network.util.JavaUtils.deleteRecursively(JavaUtils.java:118) at org.apache.spark.network.util.JavaUtils.deleteRecursivelyUsingJavaIO(JavaUtils.java:128) at org.apache.spark.network.util.JavaUtils.deleteRecursively(JavaUtils.java:118) at org.apache.spark.network.util.JavaUtils.deleteRecursivelyUsingJavaIO(JavaUtils.java:128) at org.apache.spark.network.util.JavaUtils.deleteRecursively(JavaUtils.java:118) at org.apache.spark.network.util.JavaUtils.deleteRecursively(JavaUtils.java:91) at org.apache.spark.util.Utils$.deleteRecursively(Utils.scala:1141) at org.apache.spark.util.ShutdownHookManager$.$anonfun$new$4(ShutdownHookManager.scala:65) at org.apache.spark.util.ShutdownHookManager$.$anonfun$new$4$adapted(ShutdownHookManager.scala:62) at scala.collection.IndexedSeqOptimized.foreach(IndexedSeqOptimized.scala:36) at scala.collection.IndexedSeqOptimized.foreach$(IndexedSeqOptimized.scala:33) at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:198) at org.apache.spark.util.ShutdownHookManager$.$anonfun$new$2(ShutdownHookManager.scala:62) at org.apache.spark.util.SparkShutdownHook.run(ShutdownHookManager.scala:214) at org.apache.spark.util.SparkShutdownHookManager.$anonfun$runAll$2(ShutdownHookManager.scala:188) at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23) at org.apache.spark.util.Utils$.logUncaughtExceptions(Utils.scala:1996) at org.apache.spark.util.SparkShutdownHookManager.$anonfun$runAll$1(ShutdownHookManager.scala:188) at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23) at scala.util.Try$.apply(Try.scala:213) at org.apache.spark.util.SparkShutdownHookManager.runAll(ShutdownHookManager.scala:188) at org.apache.spark.util.SparkShutdownHookManager$$anon$2.run(ShutdownHookManager.scala:178) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 21/10/24 19:56:20 INFO ShutdownHookManager: Deleting directory C:\Users\Dell\AppData\Local\Temp\spark-48a006a1-5598-4b8e-8309-dcbbaa6c3914\userFiles-9fb96294-452c-4191-9d5f-162c43c4fecf 21/10/24 19:56:20 ERROR ShutdownHookManager: Exception while deleting Spark temp dir: C:\Users\Dell\AppData\Local\Temp\spark-48a006a1-5598-4b8e-8309-dcbbaa6c3914\userFiles-9fb96294-452c-4191-9d5f-162c43c4fecf java.io.IOException: Failed to delete: C:\Users\Dell\AppData\Local\Temp\spark-48a006a1-5598-4b8e-8309-dcbbaa6c3914\userFiles-9fb96294-452c-4191-9d5f-162c43c4fecf\microsoft-spark-3-1_2.12-2.0.0.jar at org.apache.spark.network.util.JavaUtils.deleteRecursivelyUsingJavaIO(JavaUtils.java:144) at org.apache.spark.network.util.JavaUtils.deleteRecursively(JavaUtils.java:118) at org.apache.spark.network.util.JavaUtils.deleteRecursivelyUsingJavaIO(JavaUtils.java:128) at org.apache.spark.network.util.JavaUtils.deleteRecursively(JavaUtils.java:118) at org.apache.spark.network.util.JavaUtils.deleteRecursively(JavaUtils.java:91) at org.apache.spark.util.Utils$.deleteRecursively(Utils.scala:1141) at org.apache.spark.util.ShutdownHookManager$.$anonfun$new$4(ShutdownHookManager.scala:65) at org.apache.spark.util.ShutdownHookManager$.$anonfun$new$4$adapted(ShutdownHookManager.scala:62) at scala.collection.IndexedSeqOptimized.foreach(IndexedSeqOptimized.scala:36) at scala.collection.IndexedSeqOptimized.foreach$(IndexedSeqOptimized.scala:33) at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:198) at org.apache.spark.util.ShutdownHookManager$.$anonfun$new$2(ShutdownHookManager.scala:62) at org.apache.spark.util.SparkShutdownHook.run(ShutdownHookManager.scala:214) at org.apache.spark.util.SparkShutdownHookManager.$anonfun$runAll$2(ShutdownHookManager.scala:188) at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23) at org.apache.spark.util.Utils$.logUncaughtExceptions(Utils.scala:1996) at org.apache.spark.util.SparkShutdownHookManager.$anonfun$runAll$1(ShutdownHookManager.scala:188) at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23) at scala.util.Try$.apply(Try.scala:213) at org.apache.spark.util.SparkShutdownHookManager.runAll(ShutdownHookManager.scala:188) at org.apache.spark.util.SparkShutdownHookManager$$anon$2.run(ShutdownHookManager.scala:178) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)

________________________________ From: Terry Kim ***@***.***> Sent: Sunday, October 24, 2021 7:53 PM To: dotnet/spark ***@***.***> Cc: visguru1970 ***@***.***>; Mention ***@***.***> Subject: Re: [dotnet/spark] issue in saving the dataframe to a local file (Discussion #986) Can you attach a full log (not just exception)? - You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub<https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fdotnet%2Fspark%2Fdiscussions%2F986%23discussioncomment-1529080&data=04%7C01%7C%7C629bd1b164f34f5a673f08d997497637%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637707164031316149%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=9u5AUr2a4BhBAycavL7yREV57qYqTVAwitd68Nj6Gdo%3D&reserved=0>, or unsubscribe<https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAVUWCLGZQBZHDMKOMWHV5TLUISL7DANCNFSM5GRH3XZQ&data=04%7C01%7C%7C629bd1b164f34f5a673f08d997497637%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637707164031326105%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=u5Re0lmL7GQExM4sTwo5V4lbIEj6U%2B9%2Ft9G5qDKCyJI%3D&reserved=0>. Triage notifications on the go with GitHub Mobile for iOS<https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fapps.apple.com%2Fapp%2Fapple-store%2Fid1477376905%3Fct%3Dnotification-email%26mt%3D8%26pt%3D524675&data=04%7C01%7C%7C629bd1b164f34f5a673f08d997497637%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637707164031326105%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=BKA8qxfoL8lX0PlpwPcvFDxyTjMioPO1ngw642SXilc%3D&reserved=0> or Android<https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fplay.google.com%2Fstore%2Fapps%2Fdetails%3Fid%3Dcom.github.android%26referrer%3Dutm_campaign%253Dnotification-email%2526utm_medium%253Demail%2526utm_source%253Dgithub&data=04%7C01%7C%7C629bd1b164f34f5a673f08d997497637%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637707164031326105%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=vQNCBTk6l3cspTq2Jc%2BRORvY020qe0A6ox%2BgbBe3v74%3D&reserved=0>.

[imback82]

Caused by: java.lang.UnsatisfiedLinkError: org.apache.hadoop.io.nativeio.NativeIO$Windows.access0(Ljava/lang/String;I)Z
        at org.apache.hadoop.io.nativeio.NativeIO$Windows.access0(Native Method)
        at org.apache.hadoop.io.nativeio.NativeIO$Windows.access(NativeIO.java:645)

Two things you can try:

1. Download winutils for your Hadoop version from https://github.com/steveloughran/winutils and use it. Or
2. Try 2.7 Hadoop prebuilt for your Spark distro

[visguru1970]

Hello Terry, The issue is fixed now. we need to download the hadoop.dll https://github.com/cdarlint/winutils/blob/master/hadoop-3.2.1/bin/hadoop.dll<https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fcdarlint%2Fwinutils%2Fblob%2Fmaster%2Fhadoop-3.2.1%2Fbin%2Fhadoop.dll&data=04%7C01%7C%7C415986fd3fe44555d06f08d997dc6a5b%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637707795191483377%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=E3jvm6p3laPLBGrPkGO%2BG%2FRyebUM6Xxdrq%2FYQXsWlpA%3D&reserved=0> we already have winutils.exe Thanks

…

________________________________ From: Terry Kim ***@***.***> Sent: Sunday, October 24, 2021 8:07 PM To: dotnet/spark ***@***.***> Cc: visguru1970 ***@***.***>; Mention ***@***.***> Subject: Re: [dotnet/spark] issue in saving the dataframe to a local file (Discussion #986) Caused by: java.lang.UnsatisfiedLinkError: org.apache.hadoop.io.nativeio.NativeIO$Windows.access0(Ljava/lang/String;I)Z at org.apache.hadoop.io.nativeio.NativeIO$Windows.access0(Native Method) at org.apache.hadoop.io.nativeio.NativeIO$Windows.access(NativeIO.java:645) Two things you can try: 1. Download winutils for your Hadoop version and use it. Or 2. Try 2.7 Hadoop prebuilt for your Spark distro — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub<https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fdotnet%2Fspark%2Fdiscussions%2F986%23discussioncomment-1529103&data=04%7C01%7C%7C5696f5863bb3455b1a2e08d9974b7a50%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637707172687784391%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=9m4J2X5WobAktqq9TbqdYtw3BU0oMddMk2R%2BPf%2FRh4Q%3D&reserved=0>, or unsubscribe<https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAVUWCLBBFMOF2JM46Z24XOTUISNVHANCNFSM5GRH3XZQ&data=04%7C01%7C%7C5696f5863bb3455b1a2e08d9974b7a50%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637707172687794341%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=ZWtbv9SgyI%2BtoDxrKr%2BmMiJXXhYShW82mZHLZAsmcn4%3D&reserved=0>. Triage notifications on the go with GitHub Mobile for iOS<https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fapps.apple.com%2Fapp%2Fapple-store%2Fid1477376905%3Fct%3Dnotification-email%26mt%3D8%26pt%3D524675&data=04%7C01%7C%7C5696f5863bb3455b1a2e08d9974b7a50%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637707172687794341%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=dSBCjantYfHjOVvXUKnwv4dZHtR7Kbg%2FQMWJLER7itY%3D&reserved=0> or Android<https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fplay.google.com%2Fstore%2Fapps%2Fdetails%3Fid%3Dcom.github.android%26referrer%3Dutm_campaign%253Dnotification-email%2526utm_medium%253Demail%2526utm_source%253Dgithub&data=04%7C01%7C%7C5696f5863bb3455b1a2e08d9974b7a50%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637707172687804298%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=vMoqCBsEl5DJrlEv29g46LZhzuPK%2BjzL5QtzNeKoM2I%3D&reserved=0>.