Add read_<format> functions to the etna.datasets module for loading TSDataset directly from files.

[ArtemLiA]

Description

Hi! I suggest to add to the etna.datasets module functions read_<format> , where format = csv, excel and other popular file formats for storing tabular data, for reading files into TSDataset objects. What do you think about that?

Thoughts

It would be perfect to add support of all formats available in pandas. But as first three it seem to me optimal to add support of reading of csv, excel and sql formats.

Related

No response

[d-a-bunin]

Hi! I'm not sure that we should support 3 formats for loading data from disk and I'm not sure that these formats should be human-readable.

But I agree that we probably need 2 things:

• Ability to save dataset on disk using some format. We could think how to store all the inner components of a dataset into some kind of archive like we do for pipelines for example.
• Ability to load saved dataset from disk.

This format could be different from the format we use in load_datasets I suppose.

[ArtemLiA]

Do you want to store TSDataset object as several separated files or in a single archive/folder?

[d-a-bunin]

I suggest that this could be an archive that contains several files inside.

[ArtemLiA]

Let's implement the possibility of storing TSDataset objects in zip-archieve, where fields df and df_exog will be stored as two different files in any format supported by pandas (csv, sql, excel, e.t.c.) and other fields of object will be stored in json file. So, I suggest to implement the functions to_zip and read_zip, which will save and read TSDataset objects into above format respectively.

What do you think?

[d-a-bunin]

I don't agree that we should deviate from API for saving all other objects in the library. The method for storing should be named save, the method for loading should be named load.

I'm also not very sure that it will be easy to save the rest of the object into json file, because some inner attributes of TSDataset aren't json-serializable and some processing should be done in order to save them into json.

[ArtemLiA]

We could use pickle to store the rest of the TSDataset object, but this technology has serious security issues.

As I understand, the main diffuculties are related to the storage of HierarchicalStructure object. Maybe we could write custom json-encoder and decoder for these object?