Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

panic when import a large number of files #95

Closed
IanSmith123 opened this issue Oct 29, 2020 · 0 comments
Closed

panic when import a large number of files #95

IanSmith123 opened this issue Oct 29, 2020 · 0 comments
Labels
process/fixed Process of bug type/bug Type: something is unexpected

Comments

@IanSmith123
Copy link
Contributor

question mentioned at https://discuss.nebula-graph.com.cn/t/topic/1579/21?u=les1ie

If I try to import a larger number of files, eg: 1000 csv files, with a yaml file more than 10000 lines, nebula_importer will panic error

How to reproduce:

  1. ganerate sample csv files
from pathlib import Path
import os

dump_dir = Path('./dump')
if not os.path.exists(dump_dir):
    os.mkdir(dump_dir)


def generate_csv():
    num = 10000
    for i in range(num):
        with open(f'{dump_dir}/vertex_{i}.csv', 'w') as f:
            f.write("123\n")


generate_csv()
  1. download error config.yaml
    out.zip

  2. import csv

python3 reproduce.py
cp path_to_nebula_impoerter_exec dump/
cd dump
./nebula_importer -c out.yaml
  1. screenshot
    image
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
process/fixed Process of bug type/bug Type: something is unexpected
Projects
None yet
Development

No branches or pull requests

2 participants