[Bug] cpu not fully used, data_time load slow

[YAwei666]

Branch

main branch (mmpretrain version)

Describe the bug

python tools/train.py configs/resnet/resnet50_8xb32_in1k_2.py
base = [
'../base/models/resnet50.py', '../base/datasets/imagenet_bs32.py',
'../base/schedules/imagenet_bs256_coslr.py', '../base/default_runtime.py'
]
model = dict(
backbone=dict(
frozen_stages=2,
init_cfg=dict(
type='Pretrained',
checkpoint='https://download.openmmlab.com/mmclassification/v0/resnet/resnet50_8xb32_in1k_20210831-ea4938fc.pth',
prefix='backbone',
)),
head=dict(num_classes=5),
)

>>>>>>>>>>>>>>> 在这里重载数据配置 >>>>>>>>>>>>>>>>>>>

data_root = '/mnt/data//dataset'
train_dataloader = dict(
batch_size=192,
dataset=dict(
type='CustomDataset',
data_root=data_root,
ann_file='meta/train.txt', # 我们假定使用子文件夹格式，因此需要将标注文件置空
data_prefix='',
))
val_dataloader = dict(
batch_size=192,
dataset=dict(
type='CustomDataset',
data_root=data_root,
ann_file='meta/test.txt', # 我们假定使用子文件夹格式，因此需要将标注文件置空
data_prefix='',
))
test_dataloader = val_dataloader

optim_wrapper = dict(
optimizer=dict(type='SGD', lr=0.01, momentum=0.9, weight_decay=0.0001))

# 学习率策略

param_scheduler = dict(

type='MultiStepLR', by_epoch=True, milestones=[15], gamma=0.1)

train, val, test setting

train_cfg = dict(by_epoch=True, max_epochs=30, val_interval=1)

'../base/models/resnet50.py'

model settings

model = dict(
type='ImageClassifier',
backbone=dict(
type='ResNeSt',
depth=50,
num_stages=4,
out_indices=(3, ),
style='pytorch'),
neck=dict(type='GlobalAveragePooling'),
head=dict(
type='LinearClsHead',
num_classes=1000,
in_channels=2048,
loss=dict(
type='LabelSmoothLoss',
label_smooth_val=0.1,
num_classes=1000,
reduction='mean',
loss_weight=1.0),
topk=(1, 5),
cal_acc=False),
train_cfg=dict(augments=dict(type='Mixup', alpha=0.2)),
)

'../base/datasets/imagenet_bs32.py'

dataset settings

dataset_type = 'ImageNet'
data_preprocessor = dict(
num_classes=1000,
# RGB format normalization parameters
mean=[123.675, 116.28, 103.53],
std=[58.395, 57.12, 57.375],
# convert image from BGR to RGB
to_rgb=True,
)

train_pipeline = [
dict(type='LoadImageFromFile',imdecode_backend='pillow' ),
dict(type='RandomResizedCrop', scale=224),
dict(type='RandomFlip', prob=0.5, direction='horizontal'),
dict(type='PackInputs'),
]

test_pipeline = [
dict(type='LoadImageFromFile',imdecode_backend='pillow'),
dict(type='ResizeEdge', scale=256, edge='short'),
dict(type='CenterCrop', crop_size=224),
dict(type='PackInputs'),
]

train_dataloader = dict(
batch_size=128,
num_workers=12,
dataset=dict(
type=dataset_type,
data_root='data/imagenet',
pipeline=train_pipeline),
sampler=dict(type='DefaultSampler', shuffle=True),
)

val_dataloader = dict(
batch_size=128,
num_workers=12,
dataset=dict(
type=dataset_type,
data_root='data/imagenet',
pipeline=test_pipeline),
sampler=dict(type='DefaultSampler', shuffle=False),
)
val_evaluator = dict(type='Accuracy', topk=(1))

If you want standard test, please manually configure the test dataset

test_dataloader = val_dataloader
test_evaluator = val_evaluator

'../base/schedules/imagenet_bs256_coslr.py',

optimizer

optim_wrapper = dict(
optimizer=dict(type='SGD', lr=0.8, momentum=0.9, weight_decay=5e-5))

learning policy

param_scheduler = [
dict(type='LinearLR', start_factor=0.1, by_epoch=True, begin=0, end=5),
dict(type='CosineAnnealingLR', T_max=95, by_epoch=True, begin=5, end=100)
]

train, val, test setting

train_cfg = dict(by_epoch=True, max_epochs=100, val_interval=1)
val_cfg = dict()
test_cfg = dict()

NOTE: auto_scale_lr is for automatically scaling LR,

based on the actual training batch size.

auto_scale_lr = dict(base_batch_size=1024)

lr: 1.0000e-02 eta: 17:43:57 time: 3.5791 data_time: 3.3401 memory: 4676 loss: 0.3175

Environment

{'sys.platform': 'linux',
'Python': '3.8.19 (default, Mar 20 2024, 19:58:24) [GCC 11.2.0]',
'CUDA available': True,
'MUSA available': False,
'numpy_random_seed': 2147483648,
'GPU 0': 'NVIDIA GeForce RTX 3090',
'CUDA_HOME': ':/usr/local/cuda',
'GCC': 'gcc (Ubuntu 5.4.0-6ubuntu1~16.04.12) 5.4.0 20160609',
'PyTorch': '1.10.1',
'TorchVision': '0.11.2',
'OpenCV': '4.10.0',
'MMEngine': '0.10.4',
'MMCV': '2.2.0',
'MMPreTrain': '1.2.0+'}

Other information

No response

[YAwei666]

it seems only 2 cpu kernels works. 怎么回事呢

[liuwake]

This is strange. Based on your information, it can be seen that you have successfully started 12 num_workers . However, only two CPU threads are occupied. Does your server have virtualization technology enabled, which may result in you only being able to use two CPU threads