Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

文件读取问题 #163

Open
DoraTiger opened this issue Jul 27, 2021 · 2 comments
Open

文件读取问题 #163

DoraTiger opened this issue Jul 27, 2021 · 2 comments

Comments

@DoraTiger
Copy link

目前在拜读源码的时候，从文件加载缓存的过程是通过openMP进行并行加速的，这样的话，如果是多节点运行，那每个节点在加载缓存阶段应该都会完整的读取一份数据，这样就会超出预估的内存使用上限了吧。
@ustcyu
Copy link
Contributor

ustcyu commented Aug 14, 2021

每个节点只加载自己负责的部分数据。

@luoyeguigen001
Copy link

多节点读写hdfs是不是会有问题

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@DoraTiger @luoyeguigen001 @ustcyu