Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Bug] [Connector-V2] HiveSource Connector read orc file format table error in Spark Engine #2837

Closed
2 of 3 tasks
Tracked by #2793
EricJoy2048 opened this issue Sep 21, 2022 · 0 comments · Fixed by #2845
Closed
2 of 3 tasks
Tracked by #2793
Assignees
Labels

Comments

@EricJoy2048
Copy link
Member

Search before asking

  • I had searched in the issues and found no similar issues.

What happened

The test table schema and test data can found in #2793.

Test job error.

SeaTunnel Version

dev

SeaTunnel Config

env {
  # You can set flink configuration here
  source.parallelism = 3
  job.name="test_hiveorc_source_to_console"
}

source {
  # This is a example input plugin **only for test and demonstrate the feature input plugin**

  Hive {
    table_name = "test_hive.test_hive_source_orc"
    metastore_uri = "thrift://ctyun7:9083"
  }

  # If you would like to get more information about how to configure seatunnel and see full list of input plugins,
  # please go to https://seatunnel.apache.org/docs/flink/configuration/source-plugins/Fake
}

transform {

}

sink {
  # choose stdout output plugin to output data to console
  Console {
  }

  # If you would like to get more information about how to configure seatunnel and see full list of output plugins,
  # please go to https://seatunnel.apache.org/docs/flink/configuration/sink-plugins/Console
}

Running Command

sh start-seatunnel-spark-connector-v2.sh --config ../config/spark_hiveorc_to_console.conf --deploy-mode client --master local

Error Exception

22/09/21 21:07:54 INFO hive.metastore: Trying to connect to metastore with URI thrift://ctyun7:9083
22/09/21 21:07:54 INFO hive.metastore: Connected to metastore.
22/09/21 21:07:55 INFO impl.OrcCodecPool: Got brand-new codec ZLIB
Exception in thread "main" java.lang.NoSuchMethodError: org.apache.orc.Reader.close()V
        at org.apache.seatunnel.connectors.seatunnel.file.source.reader.OrcReadStrategy.getSeaTunnelRowTypeInfo(OrcReadStrategy.java:125)
        at org.apache.seatunnel.connectors.seatunnel.file.hdfs.source.HdfsFileSource.prepare(HdfsFileSource.java:70)
        at org.apache.seatunnel.connectors.seatunnel.hive.source.HiveSource.prepare(HiveSource.java:82)
        at org.apache.seatunnel.core.starter.spark.execution.SourceExecuteProcessor.initializePlugins(SourceExecuteProcessor.java:78)
        at org.apache.seatunnel.core.starter.spark.execution.AbstractPluginExecuteProcessor.<init>(AbstractPluginExecuteProcessor.java:49)
        at org.apache.seatunnel.core.starter.spark.execution.SourceExecuteProcessor.<init>(SourceExecuteProcessor.java:49)
        at org.apache.seatunnel.core.starter.spark.execution.SparkExecution.<init>(SparkExecution.java:48)
        at org.apache.seatunnel.core.starter.spark.command.SparkApiTaskExecuteCommand.execute(SparkApiTaskExecuteCommand.java:53)
        at org.apache.seatunnel.core.starter.Seatunnel.run(Seatunnel.java:40)
        at org.apache.seatunnel.core.starter.spark.SeatunnelSpark.main(SeatunnelSpark.java:34)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
        at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:849)
        at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:167)
        at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:195)
        at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:86)
        at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:924)
        at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:933)
        at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
22/09/21 21:07:56 INFO spark.SparkContext: Invoking stop() from shutdown hook
22/09/21 21:07:56 INFO server.AbstractConnector: Stopped Spark@66c9697e{HTTP/1.1,[http/1.1]}{0.0.0.0:4040}
22/09/21 21:07:56 INFO ui.SparkUI: Stopped Spark web UI at http://ctyun9:4040
22/09/21 21:07:56 INFO spark.MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped!
22/09/21 21:07:56 INFO memory.MemoryStore: MemoryStore cleared
22/09/21 21:07:56 INFO storage.BlockManager: BlockManager stopped
22/09/21 21:07:56 INFO storage.BlockManagerMaster: BlockManagerMaster stopped
22/09/21 21:07:56 INFO scheduler.OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped!
22/09/21 21:07:56 INFO spark.SparkContext: Successfully stopped SparkContext
22/09/21 21:07:56 INFO util.ShutdownHookManager: Shutdown hook called
22/09/21 21:07:56 INFO util.ShutdownHookManager: Deleting directory /tmp/spark-c7c49a7f-014d-4ad9-b0a4-a1ee61b00457
22/09/21 21:07:56 INFO util.ShutdownHookManager: Deleting directory /tmp/spark-bf3a9487-0333-412f-8b30-ff899ec46ec8

Flink or Spark Version

Spark version: 2.4.3, scala version 2.11.12
Hadoop version: Hadoop 2.10.2

Java or Scala Version

JDK 1.8

Screenshots

No response

Are you willing to submit PR?

  • Yes I am willing to submit a PR!

Code of Conduct

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants