Multiple .h5 training files

[DanChen001]

Hi, dear Jiu, thank you for sharing the code.

Can I use multiple .h5 files to train the network? Since a single .h5 is too large to save. For caffe, we can generate multiple .h5 files to save training simples. How about pytorch？ Could you give me some suggestions?

Thanks.

[twtygqyy]

Hi @DanChen001 , for training with multiple h5 files, I think you need to modify the data generator. You can allocate memory to a certain number of patches in the initialization stage, then load images from multiple h5 files into the allocated memory repeatedly.
Let's say you have 10 h5 files and each contain 1000 images, and you create an array for 1000 images in the beginning. If you batch size is 50, then after 20 iterations, your data generator will load another h5 file into the memory, copy the images into the array, and continue the training. You can also shuffle the data every time you load the images.
I hope this will help you to implement your own data generator.

[DanChen001]

Hi @twtygqyy Sorry for the late reply. Thank you for helpful suggestions.