Adding a dataset

[Galvo87]

Hi, I would like to add my own dataset, which consists in several mixed speech/music files.
In the README you mention the need for creating two different folders, but I cannot fully understand the "repartition of the data between each set for each type of label" part...
Could you please give us an example dir tree to use as reference?
Thanks

[qlemaire22]

Hi! My data folder is divided like this:

- dataset1
     - audio
     - filelists
- dataset2
     - audio
     - filelists
...

The audio folder contains all the .wav files with their associated labels as .txt.
And the filelists folder contains the names of the audio files (without extension) dispatched into different files to create the test/validation and training set: mixed_train, mixed_val, mixed_test, music_train, music_val, speech_train, speech_val or noise_train (it is just, on line, one filename).

mixed, audio, speech or noise corresponds to the time of audio.

I hope it will help! Tell me if it is not clear enough.

[Galvo87]

Thanks! I have a lot of mixed content, I should try to gather also clean speech and music...
Btw, I still not fully understand the proper structure and matching of proper files/timestamps...
Test, validation and training set could be the same files, or must they all be different?
It would be awesome if you could provide a sample folder to examine for the noobs.
Thanks

[qlemaire22]

Mixed content is technically the best since it represents the goal you want to achieve. It is also the most expensive kind of data.
But having clean speech and music is an easy way to have a bigger dataset.

Test, validation and training must be strictly different, otherwise you will overfit the data.

I can try to give a more detailed example, if you have a custom dataset with 3 audio of mixed content:

- custom_dataset
    - audio
        audio1.wav
        audio1.txt
        audio2.wav
        audio2.txt
        audio3.wav
        audio3.txt
    - filelists
        mixed_train
        mixed_test
        mixed_validation

With mixed_train being:

audio1

With mixed_test being:

audio2

With mixed_validation being:

audio3

For the .txt files, if audio 1 contains music from 0s to 5s and speech from 2s to 10s, audio1.txt will be

0 5 speech
2 10 music

Is it clearer?

[Galvo87]

Sure, thank you!
I'll surely give it a try.

[Galvo87]

Hi, I successfully installed everything and added my own datasets, but when issuing python prepare_dataset/prepare_audio.py my_own_dataset I always get:

Traceback (most recent call last): File "prepare_dataset/prepare_audio.py", line 224, in <module> resample_dataset(args.data_location, args.dataset) File "prepare_dataset/prepare_audio.py", line 18, in resample_dataset cfg = utils.load_json('../datasets.json') File "/home/developer/Documents/speech-music-detection/smd/utils.py", line 99, in load_json with open(filename) as f: FileNotFoundError: [Errno 2] No such file or directory: '../datasets.json'

Of course dataset.json is at the project's root, and contains:

{ "my_own_dataset": { "data_folder": "my_own_dataset/audio", "filelists_folder": "my_own_dataset/filelists" } }

What am I missing?
Thanks