Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

escel文件中数据量比较多，有几十万条数据，然后容易OOM #8

Open
1015770492 opened this issue Jul 27, 2022 · 2 comments
Open

escel文件中数据量比较多，有几十万条数据，然后容易OOM #8

1015770492 opened this issue Jul 27, 2022 · 2 comments
Labels
bug Something isn't working

Comments

@1015770492
Copy link
Owner

使用ForkJoin框架进行并发导入，虽然导入的粒度可控，性能可以，但是特别耗内存。

内存是有限的，在java中将excel数据转实体的过程中，如果excel有几十万条数据，
就会构建几十万个实体，况且excel单元格如果有很长的文本，特别容易导致OOM。
@1015770492
Copy link
Owner Author

今天开始解决这个内存问题，打算采用异步分段处理大数据量问题，内存方面会根据虚拟机内存自动处理内存问题。

@1015770492 1015770492 added the bug Something isn't working label Jul 27, 2022
@1015770492
Copy link
Owner Author

解压一个42M的xlsx文件，里面有79w条数据，花费了1.4s左右。
79w条数据的xml解析遍历花费36s左右。没有内存溢出风险，因为用的是固定的缓冲大小

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@1015770492