Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fix issue #43 (empty files creation) and improve reading/writing speed #57

Merged
merged 1 commit into from
May 8, 2024
Merged

Fix issue #43 (empty files creation) and improve reading/writing speed #57

merged 1 commit into from
May 8, 2024

Commits on May 8, 2024

  1. Fix issue NVIDIA#43 (empty files creation) and improve reading/writin… 

    …g speed

This commit fixes issue NVIDIA#43 (empty files created when invoking reshard_jsonl method at nemo_curator.utils.file_utils.py) by double-checking the files size after being generated, and deleting them with size zero.

In addition to that, I have noticed there is no need to parse to JSON object the content of the different lines, which should be already in json format. By removing that extra-parsing, there is a significant speed up in the execution of this method.

Signed-off-by: Miguel Martínez <[email protected]>
    @miguelusque
    miguelusque authored May 8, 2024
    Configuration menu
    Copy the full SHA
    a3325ec View commit details
    Browse the repository at this point in the history