Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Bug] [Flink Jar] Download hdfs jar fail when running flink jar task:xxxx is not a valid DFS filename #1098

Open
3 tasks done
weaksloth opened this issue Jul 4, 2023 · 3 comments
Labels
bug Something isn't working

Comments

@weaksloth
Copy link
Contributor

Search before asking

  • I had searched in the issues and found no similar issues.

What happened

When running flink jar task with a resource which is stored in hdfs,taier can not download jar from hdfs successfully.And it makes flink jobs run fail.
图片

What you expected to happen

I have seen the source code and debug,found that the reason for this problem is related to StreamSqlFormatService#generateAddJarSql(String,String) and FileSystemManager#downloadFile(...)

StreamSqlFormatService will replace hdfs://localhost:9820 to hdfs:/localhost:9820,

When I return resource url directly, here comes the second question:
FileSystemManager will remove prefix(hdfs://), which make url to localhost:9820.Of course,HdfsFileManage download fail

How to reproduce

follow below steps can produce this problem:

  1. upload a flink application jar and choose hdfs storage
    resource-info
  2. create flink jar task and run.

Anything else

No response

Version

master

Are you willing to submit PR?

  • Yes I am willing to submit a PR!

Code of Conduct

@weaksloth weaksloth added the bug Something isn't working label Jul 4, 2023
@weaksloth
Copy link
Contributor Author

@vainhope 请教下大佬为什么要去将//替换掉,是出于什么原因考虑的呢?此外,对于hdfs前缀的去除是不是没有考虑完善,只去除了hdfs://,实际上还可能会有ip/host:port信息没有被去除

@vainhope
Copy link
Collaborator

vainhope commented Jul 4, 2023

flink的udf 资源目前都是仅支持sftp
hdfs的支持暂时还没开发
这个问题是前端入口未屏蔽导致的

@weaksloth
Copy link
Contributor Author

了解了大佬

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

2 participants