Skip to content

Commit

Permalink
Update docs: Remove DataModuleConfig (#3841)
Browse files Browse the repository at this point in the history
* Update DataModuleConfig docs

* Fix typo
  • Loading branch information
harimkang authored Aug 16, 2024
1 parent d77423e commit ac61247
Show file tree
Hide file tree
Showing 16 changed files with 64 additions and 81 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -152,14 +152,12 @@ To use this feature, add the following parameter:

.. code-block:: python
from otx.core.config.data import DataModuleConfig
from otx.core.data.module import OTXDataModule
data_config = DataModuleConfig(..., auto_num_workers=True)
datamodule = OTXDataModule(..., config=data_config)
datamodule = OTXDataModule(..., auto_num_workers=True)
.. tab-item:: CLI

.. code-block:: shell
(otx) ...$ otx train ... --data.config.auto_num_workers True
(otx) ...$ otx train ... --data.auto_num_workers True
Original file line number Diff line number Diff line change
Expand Up @@ -25,9 +25,9 @@ For default setting, the square root of (number of old data/number of new data)
.. code-block:: shell
(otx) ...$ otx train ... \
--data.config.train_subset.sampler.class_path otx.algo.samplers.class_incremental_sampler.ClassIncrementalSampler \
--data.config.train_subset.sampler.init_args.old_classes '[car,truck]' \
--data.config.train_subset.sampler.init_args.new_classes '[bus]'
--data.train_subset.sampler.class_path otx.algo.samplers.class_incremental_sampler.ClassIncrementalSampler \
--data.train_subset.sampler.init_args.old_classes '[car,truck]' \
--data.train_subset.sampler.init_args.new_classes '[bus]'
Balanced Sampler
Expand Down Expand Up @@ -55,4 +55,4 @@ It helps ensure balanced sampling by class based on the distribution of class la
.. code-block:: shell
(otx) ...$ otx train ... \
--data.config.train_subset.sampler.class_path otx.algo.samplers.balanced_sampler.BalancedSampler
--data.train_subset.sampler.class_path otx.algo.samplers.balanced_sampler.BalancedSampler
Original file line number Diff line number Diff line change
Expand Up @@ -27,14 +27,12 @@ training time in those cases.

.. code-block:: python
from otx.core.config.data import DataModuleConfig
from otx.core.data.module import OTXDataModule
data_config = DataModuleConfig(..., mem_cache_size="8GB")
datamodule = OTXDataModule(..., config=data_config)
datamodule = OTXDataModule(..., mem_cache_size="8GB")
.. tab-item:: CLI

.. code-block:: shell
(otx) ...$ otx train ... --data.config.mem_cache_size 8GB
(otx) ...$ otx train ... --data.mem_cache_size 8GB
46 changes: 21 additions & 25 deletions docs/source/guide/explanation/additional_features/tiling.rst
Original file line number Diff line number Diff line change
Expand Up @@ -47,25 +47,24 @@ Enable Tiling via OTX Training

Currently, tiling is supported for both detection and instance segmentation models. Please refer to :doc:`../algorithms/object_detection/object_detection` and :doc:`../algorithms/segmentation/instance_segmentation` for more details.

To enable tiling in OTX training, set ``data.config.tile_config.enable_tiler`` parameter to 1. Here's an example of enabling tiling:
To enable tiling in OTX training, set ``data.tile_config.enable_tiler`` parameter to 1. Here's an example of enabling tiling:

.. tab-set::

.. tab-item:: API

.. code-block:: python
from otx.core.config.data import DataModuleConfig, TileConfig
from otx.core.config.data import TileConfig
from otx.core.data.module import OTXDataModule
data_config = DataModuleConfig(..., tile_config=TileConfig(enable_tiler=True))
datamodule = OTXDataModule(..., config=data_config)
datamodule = OTXDataModule(..., tile_config=TileConfig(enable_tiler=True))
.. tab-item:: CLI

.. code-block:: shell
(otx) ...$ otx train ... --data.config.tile_config.enable_tiler True
(otx) ...$ otx train ... --data.tile_config.enable_tiler True
Tile Size and Tile Overlap Optimization
Expand All @@ -86,20 +85,19 @@ Here's an example of setting the object size ratio to 5%:

.. code-block:: python
from otx.core.config.data import DataModuleConfig, TileConfig
from otx.core.config.data import TileConfig
from otx.core.data.module import OTXDataModule
tile_config = TileConfig(enable_tiler=True, enable_adaptive_tiling=True, object_tile_ratio=0.05)
data_config = DataModuleConfig(..., tile_config=tile_config)
datamodule = OTXDataModule(..., config=data_config)
datamodule = OTXDataModule(..., tile_config=tile_config)
.. tab-item:: CLI

.. code-block:: shell
(otx) ...$ otx train ... --data.config.tile_config.enable_tiler True \ # enable tiling
--data.config.tile_config.enable_adaptive_tiling True \ # enable automatic tiling parameter optimization
--data.config.tile_config.object_tile_ratio 0.05 # set the object size ratio to 5%
(otx) ...$ otx train ... --data.tile_config.enable_tiler True \ # enable tiling
--data.tile_config.enable_adaptive_tiling True \ # enable automatic tiling parameter optimization
--data.tile_config.object_tile_ratio 0.05 # set the object size ratio to 5%
After determining the tile size, the tile overlap is computed by dividing the largest object size in the training dataset by the adaptive tile size.
Expand All @@ -116,28 +114,27 @@ Since training and validation on all tiles from a high-resolution image dataset

It's important to note that sampling is applied to the training and validation datasets, not the test dataset.

This can be configured with ``data.config.tile_config.enable_adaptive_tiling`` parameter. Here's an example:
This can be configured with ``data.tile_config.enable_adaptive_tiling`` parameter. Here's an example:

.. tab-set::

.. tab-item:: API

.. code-block:: python
from otx.core.config.data import DataModuleConfig, TileConfig
from otx.core.config.data import TileConfig
from otx.core.data.module import OTXDataModule
tile_config = TileConfig(enable_tiler=True, enable_adaptive_tiling=True, sampling_ratio=0.5)
data_config = DataModuleConfig(..., tile_config=tile_config)
datamodule = OTXDataModule(..., config=data_config)
datamodule = OTXDataModule(..., tile_config=tile_config)
.. tab-item:: CLI

.. code-block:: shell
(otx) ...$ otx train ... --data.config.tile_config.enable_tiler True
--data.config.tile_config.enable_adaptive_tiling True
--data.config.tile_config.sampling_ratio 0.5
(otx) ...$ otx train ... --data.tile_config.enable_tiler True
--data.tile_config.enable_adaptive_tiling True
--data.tile_config.sampling_ratio 0.5
Manual Tiling Parameter Configuration
Expand All @@ -151,21 +148,20 @@ Users can disable adaptive tiling and customize the tiling process by setting th

.. code-block:: python
from otx.core.config.data import DataModuleConfig, TileConfig
from otx.core.config.data import TileConfig
from otx.core.data.module import OTXDataModule
tile_config = TileConfig(enable_tiler=True, enable_adaptive_tiling=False, tile_size=(512,512), tile_overlap=0.2)
data_config = DataModuleConfig(..., tile_config=tile_config)
datamodule = OTXDataModule(..., config=data_config)
datamodule = OTXDataModule(..., tile_config=tile_config)
.. tab-item:: CLI

.. code-block:: shell
(otx) ...$ otx train ... --data.config.tile_config.enable_tiler True
--data.config.tile_config.enable_adaptive_tiling False
--data.config.tile_config.tile_size '[512,512]'
--data.config.tile_config.tile_overlap 0.2
(otx) ...$ otx train ... --data.tile_config.enable_tiler True
--data.tile_config.enable_adaptive_tiling False
--data.tile_config.tile_size '[512,512]'
--data.tile_config.tile_overlap 0.2
By specifying these parameters, automatic tiling parameter optimization is disabled, and the tile size is configured to 512x512 pixels with a 10% overlap between tiles.

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -159,7 +159,7 @@ You can select the model configuration and run Semi-SL training with the command
(otx) ...$ otx train \
--config {semi_sl_config_path} \
--data_root {path_to_labeled_data} \
--data.config.unlabeled_subset.data_root {path_to_unlabeled_data}
--data.unlabeled_subset.data_root {path_to_unlabeled_data}
Below are the results of comparing supervised learning and semi-supervised learning for each label per class on three datasets, three models.

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -93,5 +93,5 @@ According to datasets, ``learning rate`` and ``batch size`` can be adjusted like
(otx) ...$ otx train \
--config <model_config_path> \
--data_root <path_to_data_root> \
--data.config.train_subset.batch_size <batch_size_to_be_updated> \
--data.train_subset.batch_size <batch_size_to_be_updated> \
--optimizer.lr <learning_rate_to_be_updated>
6 changes: 3 additions & 3 deletions docs/source/guide/get_started/cli_commands.rst
Original file line number Diff line number Diff line change
Expand Up @@ -349,12 +349,12 @@ The results will be saved in ``./otx-workspace/`` folder by default. The output
You also can visualize the training using ``Tensorboard`` as these logs are located in ``<work_dir>/tensorboard``.
.. note::
``--data.config.mem_cache_size`` provides in-memory caching for decoded images in main memory.
``--data.mem_cache_size`` provides in-memory caching for decoded images in main memory.
If the batch size is large, such as for classification tasks, or if your dataset contains high-resolution images,
image decoding can account for a non-negligible overhead in data pre-processing.
This option can be useful for maximizing GPU utilization and reducing model training time in those cases.
If your machine has enough main memory, we recommend increasing this value as much as possible.
For example, you can cache approximately 10,000 of ``500x375~500x439`` sized images with ``--data.config.mem_cache_size 8GB``.
For example, you can cache approximately 10,000 of ``500x375~500x439`` sized images with ``--data.mem_cache_size 8GB``.
It is also possible to start training by omitting the recipe and just passing the paths to dataset roots, then the :doc:`auto-configuration <../explanation/additional_features/auto_configuration>` will be enabled. Based on the dataset, OpenVINO™ Training Extensions will choose the task type and recipe with the best accuracy/speed trade-off.
Expand All @@ -363,7 +363,7 @@ For example, that is how you can change the max epochs and the batch size for th
.. code-block:: shell
(otx) ...$ otx train ... --data.config.train_subset.batch_size <batch-size> --max_epochs <max-epochs>
(otx) ...$ otx train ... --data.train_subset.batch_size <batch-size> --max_epochs <max-epochs>
.. note::
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -127,7 +127,7 @@ Training
(otx) ...$ otx train \
--config src/otx/recipe/classification/multi_class_cls/mobilenet_v3_large_semisl.yaml \
--data_root data/flower_photos/labeled \
--data.config.unlabeled_subset.data_root data/flower_photos/unlabeled
--data.unlabeled_subset.data_root data/flower_photos/unlabeled
.. tab-item:: API (from_config)

Expand All @@ -137,7 +137,7 @@ Training
data_root = "data/flower_photos"
recipe = "src/otx/recipe/classification/multi_class_cls/mobilenet_v3_large_semisl.yaml"
overrides = {"data.config.unlabeled_subset.data_root": "data/flower_photos/unlabeled"}
overrides = {"data.unlabeled_subset.data_root": "data/flower_photos/unlabeled"}
engine = Engine.from_config(
config_path=recipe,
Expand All @@ -152,12 +152,11 @@ Training

.. code-block:: python
from otx.core.config.data import DataModuleConfig, UnlabeledDataConfig
from otx.core.config.data import UnlabeledDataConfig
from otx.core.data.module import OTXDataModule
from otx.engine import Engine
data_config = DataModuleConfig(..., unlabeled_subset=UnlabeledDataConfig(data_root="data/flower_photos/unlabeled", ...))
datamodule = OTXDataModule(..., config=data_config)
datamodule = OTXDataModule(..., unlabeled_subset=UnlabeledDataConfig(data_root="data/flower_photos/unlabeled", ...))
engine = Engine(..., datamodule=datamodule)
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -160,19 +160,18 @@ For example, to decrease the batch size to 4, fix the number of epochs to 100, e

.. code-block:: shell
(otx) ...$ otx train ... --data.config.train_subset.batch_size 4 \
(otx) ...$ otx train ... --data.train_subset.batch_size 4 \
--max_epochs 100
.. tab-item:: API

.. code-block:: python
from otx.core.config.data import DataModuleConfig, SubsetConfig
from otx.core.config.data import SubsetConfig
from otx.core.data.module import OTXDataModule
from otx.engine import Engine
data_config = DataModuleConfig(..., train_subset=SubsetConfig(..., batch_size=4))
datamodule = OTXDataModule(..., config=data_config)
datamodule = OTXDataModule(..., train_subset=SubsetConfig(..., batch_size=4))
engine = Engine(..., datamodule=datamodule)
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -153,19 +153,18 @@ For example, to decrease the batch size to 4, fix the number of epochs to 100, e

.. code-block:: shell
(otx) ...$ otx train ... --data.config.train_subset.batch_size 4 \
(otx) ...$ otx train ... --data.train_subset.batch_size 4 \
--max_epochs 100
.. tab-item:: API

.. code-block:: python
from otx.core.config.data import DataModuleConfig, SubsetConfig
from otx.core.config.data import SubsetConfig
from otx.core.data.module import OTXDataModule
from otx.engine import Engine
data_config = DataModuleConfig(..., train_subset=SubsetConfig(..., batch_size=4))
datamodule = OTXDataModule(..., config=data_config)
datamodule = OTXDataModule(..., train_subset=SubsetConfig(..., batch_size=4))
engine = Engine(..., datamodule=datamodule)
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -183,19 +183,18 @@ For example, to decrease the batch size to 4, fix the number of epochs to 100, e

.. code-block:: shell
(otx) ...$ otx train ... --data.config.train_subset.batch_size 4 \
(otx) ...$ otx train ... --data.train_subset.batch_size 4 \
--max_epochs 100
.. tab-item:: API

.. code-block:: python
from otx.core.config.data import DataModuleConfig, SubsetConfig
from otx.core.config.data import SubsetConfig
from otx.core.data.module import OTXDataModule
from otx.engine import Engine
data_config = DataModuleConfig(..., train_subset=SubsetConfig(..., batch_size=4))
datamodule = OTXDataModule(..., config=data_config)
datamodule = OTXDataModule(..., train_subset=SubsetConfig(..., batch_size=4))
engine = Engine(..., datamodule=datamodule)
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -251,19 +251,18 @@ For example, to decrease the batch size to 4, fix the number of epochs to 100, e

.. code-block:: shell
(otx) ...$ otx train ... --data.config.train_subset.batch_size 4 \
(otx) ...$ otx train ... --data.train_subset.batch_size 4 \
--max_epochs 100
.. tab-item:: API

.. code-block:: python
from otx.core.config.data import DataModuleConfig, SubsetConfig
from otx.core.config.data import SubsetConfig
from otx.core.data.module import OTXDataModule
from otx.engine import Engine
data_config = DataModuleConfig(..., train_subset=SubsetConfig(..., batch_size=4))
datamodule = OTXDataModule(..., config=data_config)
datamodule = OTXDataModule(..., train_subset=SubsetConfig(..., batch_size=4))
engine = Engine(..., datamodule=datamodule)
Expand Down
7 changes: 3 additions & 4 deletions docs/source/guide/tutorials/base/how_to_train/detection.rst
Original file line number Diff line number Diff line change
Expand Up @@ -263,19 +263,18 @@ For example, to decrease the batch size to 4, fix the number of epochs to 100, e

.. code-block:: shell
(otx) ...$ otx train ... --data.config.train_subset.batch_size 4 \
(otx) ...$ otx train ... --data.train_subset.batch_size 4 \
--max_epochs 100
.. tab-item:: API

.. code-block:: python
from otx.core.config.data import DataModuleConfig, SubsetConfig
from otx.core.config.data import SubsetConfig
from otx.core.data.module import OTXDataModule
from otx.engine import Engine
data_config = DataModuleConfig(..., train_subset=SubsetConfig(..., batch_size=4))
datamodule = OTXDataModule(..., config=data_config)
datamodule = OTXDataModule(..., train_subset=SubsetConfig(..., batch_size=4))
engine = Engine(..., datamodule=datamodule)
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -277,19 +277,18 @@ For example, to decrease the batch size to 4, fix the number of epochs to 100 an

.. code-block:: shell
(otx) ...$ otx train ... --data.config.train_subset.batch_size 4 \
(otx) ...$ otx train ... --data.train_subset.batch_size 4 \
--max_epochs 100
.. tab-item:: API

.. code-block:: python
from otx.core.config.data import DataModuleConfig, SubsetConfig
from otx.core.config.data import SubsetConfig
from otx.core.data.module import OTXDataModule
from otx.engine import Engine
data_config = DataModuleConfig(..., train_subset=SubsetConfig(..., batch_size=4))
datamodule = OTXDataModule(..., config=data_config)
datamodule = OTXDataModule(..., train_subset=SubsetConfig(..., batch_size=4))
engine = Engine(..., datamodule=datamodule)
Expand Down
Loading

0 comments on commit ac61247

Please sign in to comment.