Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add mux to fine-tune recipe for pruned_transducer_stateless7 #1059

Merged
merged 8 commits into from
May 17, 2023
Merged

Add mux to fine-tune recipe for pruned_transducer_stateless7 #1059

merged 8 commits into from
May 17, 2023

Conversation

yfyeung
Copy link
Collaborator

@yfyeung yfyeung commented May 15, 2023 •
edited
Loading

This PR adds Lhotse.mux to fine-tune recipe for pruned_transducer_stateless7.
For adapting already-trained models to new data, we mix 5% of the new/adaptation data with 95% of the original data by CutSet.mux in Lhotse to fine-tune.

Our observation:

  • Fine-tuning with mux can preserve the source domain's performance much better than fine-tuning directly on target data.
  • Fine-tuning with mux gets worse performance on the target domain than fine-tuning directly on target data.

Pretrained model and bpe model needed for fine-tuning:
https://huggingface.co/csukuangfj/icefall-asr-librispeech-pruned-transducer-stateless7-2022-11-11

Usage:

export CUDA_VISIBLE_DEVICES="0,1"

./pruned_transducer_stateless7/finetune.py \
  --world-size 2 \
  --num-epochs 20 \
  --start-epoch 1 \
  --exp-dir pruned_transducer_stateless7/exp_giga_finetune \
  --subset S \
  --use-fp16 1 \
  --base-lr 0.005 \
  --lr-epochs 100 \
  --lr-batches 100000 \
  --bpe-model icefall-asr-librispeech-pruned-transducer-stateless7-2022-11-11/data/lang_bpe_500/bpe.model \
  --do-finetune True \
  --use-mux True \
  --finetune-ckpt icefall-asr-librispeech-pruned-transducer-stateless7-2022-11-11/exp/pretrained.pt \
  --max-duration 500

Here are some results of adapting a model trained with LibriSpeech on GigaSpeech S:

GigaSpeech dev & test LibriSpeech test-clean & test-other config
14.26 & 14.14 2.14 & 4.87 epoch 20 avg 5
14.46 & 14.45 2.09 & 4.96 epoch 15 avg 5
14.94 & 14.86 2.18 & 5.15 epoch 10 avg 4
15.50 & 15.49 2.29 & 5.32 epoch 5 avg 2
16.59 & 16.45 2.32 & 5.57 epoch 2 avg 1

Note: we use the LR-related settings --base-lr 0.005, --lr-epochs 100, --lr-batches 100000

Baseline

Train from scratch on GigaSpeech S

GigaSpeech dev & test config
14.91 & 15.04 epoch 30 avg 14

Fine-tune from LibriSpeech

GigaSpeech dev & test config
13.59 & 13.49 epoch 11 avg 10

Note: It gets much higher WERs on LibriSpeech (about 3.1/7.1).

Full Result

dev & test sum config
14.26 & 14.14 28.40 epoch 20 avg 5
14.27 & 14.13 28.40 epoch 20 avg 7
14.28 & 14.13 28.41 epoch 20 avg 6
14.3 & 14.14 28.44 epoch 20 avg 8
14.31 & 14.16 28.47 epoch 19 avg 6
14.29 & 14.19 28.48 epoch 20 avg 4
14.31 & 14.17 28.48 epoch 19 avg 5
14.31 & 14.18 28.49 epoch 20 avg 9
14.31 & 14.19 28.50 epoch 19 avg 7
14.3 & 14.23 28.53 epoch 20 avg 3
14.32 & 14.22 28.54 epoch 20 avg 10
14.32 & 14.24 28.56 epoch 19 avg 4
14.34 & 14.23 28.57 epoch 18 avg 6
14.35 & 14.22 28.57 epoch 19 avg 8
14.32 & 14.26 28.58 epoch 19 avg 9
14.34 & 14.25 28.59 epoch 18 avg 7
14.38 & 14.22 28.60 epoch 18 avg 5
14.39 & 14.23 28.62 epoch 18 avg 4
14.37 & 14.28 28.65 epoch 18 avg 8
14.38 & 14.27 28.65 epoch 20 avg 11
14.39 & 14.28 28.67 epoch 18 avg 3
14.38 & 14.29 28.67 epoch 19 avg 3
14.35 & 14.32 28.67 epoch 20 avg 2
14.4 & 14.28 28.68 epoch 17 avg 4
14.39 & 14.29 28.68 epoch 17 avg 5
14.39 & 14.3 28.69 epoch 19 avg 10
14.4 & 14.31 28.71 epoch 17 avg 6
14.4 & 14.31 28.71 epoch 20 avg 12
14.4 & 14.32 28.72 epoch 18 avg 9
14.42 & 14.33 28.75 epoch 17 avg 7
14.4 & 14.36 28.76 epoch 17 avg 8
14.43 & 14.33 28.76 epoch 19 avg 11
14.45 & 14.32 28.77 epoch 17 avg 3
14.43 & 14.34 28.77 epoch 20 avg 13
14.4 & 14.38 28.78 epoch 19 avg 2
14.43 & 14.36 28.79 epoch 16 avg 4
14.42 & 14.37 28.79 epoch 16 avg 5
14.44 & 14.36 28.80 epoch 18 avg 10
14.45 & 14.37 28.82 epoch 16 avg 3
14.44 & 14.38 28.82 epoch 16 avg 6
14.46 & 14.36 28.82 epoch 19 avg 12
14.46 & 14.39 28.85 epoch 17 avg 2
14.47 & 14.38 28.85 epoch 17 avg 9
14.47 & 14.38 28.85 epoch 18 avg 2
14.48 & 14.37 28.85 epoch 18 avg 11
14.46 & 14.41 28.87 epoch 16 avg 7
14.49 & 14.38 28.87 epoch 20 avg 14
14.46 & 14.45 28.91 epoch 15 avg 5
14.51 & 14.41 28.92 epoch 17 avg 10
14.51 & 14.41 28.92 epoch 19 avg 13
14.49 & 14.44 28.93 epoch 15 avg 4
14.49 & 14.44 28.93 epoch 16 avg 8
14.51 & 14.44 28.95 epoch 20 avg 15
14.49 & 14.46 28.95 epoch 15 avg 6
14.54 & 14.42 28.96 epoch 16 avg 2
14.51 & 14.45 28.96 epoch 20 avg 1
14.53 & 14.44 28.97 epoch 18 avg 12
14.52 & 14.46 28.98 epoch 16 avg 9
14.48 & 14.52 29.00 epoch 14 avg 4
14.52 & 14.48 29.00 epoch 15 avg 7
14.53 & 14.48 29.01 epoch 15 avg 3
14.54 & 14.47 29.01 epoch 17 avg 11
14.57 & 14.46 29.03 epoch 19 avg 14
14.53 & 14.53 29.06 epoch 15 avg 2
14.54 & 14.53 29.07 epoch 14 avg 6
14.56 & 14.51 29.07 epoch 15 avg 8
14.58 & 14.49 29.07 epoch 20 avg 16
14.57 & 14.51 29.08 epoch 16 avg 10
14.54 & 14.55 29.09 epoch 14 avg 5
14.6 & 14.49 29.09 epoch 18 avg 13
14.61 & 14.52 29.13 epoch 17 avg 12
14.61 & 14.52 29.13 epoch 18 avg 1
14.53 & 14.6 29.13 epoch 19 avg 1
14.57 & 14.58 29.15 epoch 14 avg 3
14.6 & 14.56 29.16 epoch 16 avg 11
14.6 & 14.57 29.17 epoch 14 avg 7
14.64 & 14.53 29.17 epoch 19 avg 15
14.58 & 14.61 29.19 epoch 13 avg 3
14.63 & 14.56 29.19 epoch 15 avg 9
14.65 & 14.55 29.20 epoch 18 avg 14
14.6 & 14.61 29.21 epoch 13 avg 4
14.67 & 14.55 29.22 epoch 20 avg 17
14.61 & 14.62 29.23 epoch 13 avg 5
14.63 & 14.61 29.24 epoch 13 avg 6
14.66 & 14.6 29.26 epoch 14 avg 8
14.68 & 14.58 29.26 epoch 16 avg 1
14.68 & 14.58 29.26 epoch 17 avg 13
14.68 & 14.59 29.27 epoch 17 avg 1
14.68 & 14.59 29.27 epoch 19 avg 16
14.68 & 14.61 29.29 epoch 15 avg 10
14.69 & 14.61 29.30 epoch 16 avg 12
14.7 & 14.64 29.34 epoch 13 avg 7
14.67 & 14.67 29.34 epoch 14 avg 2
14.72 & 14.63 29.35 epoch 18 avg 15
14.71 & 14.66 29.37 epoch 14 avg 9
14.74 & 14.63 29.37 epoch 15 avg 1
14.71 & 14.67 29.38 epoch 12 avg 5
14.7 & 14.69 29.39 epoch 13 avg 2
14.74 & 14.65 29.39 epoch 20 avg 18
14.71 & 14.69 29.40 epoch 12 avg 4
14.69 & 14.72 29.41 epoch 12 avg 3
14.74 & 14.67 29.41 epoch 15 avg 11
14.76 & 14.66 29.42 epoch 17 avg 14
14.76 & 14.69 29.45 epoch 13 avg 8
14.71 & 14.75 29.46 epoch 12 avg 2
14.76 & 14.7 29.46 epoch 12 avg 6
14.75 & 14.71 29.46 epoch 14 avg 10
14.8 & 14.68 29.48 epoch 19 avg 17
14.81 & 14.7 29.51 epoch 16 avg 13
14.81 & 14.71 29.52 epoch 18 avg 16
14.75 & 14.8 29.55 epoch 14 avg 1
14.81 & 14.75 29.56 epoch 11 avg 4
14.8 & 14.76 29.56 epoch 13 avg 9
14.82 & 14.76 29.58 epoch 12 avg 7
14.81 & 14.79 29.60 epoch 11 avg 3
14.83 & 14.78 29.61 epoch 11 avg 5
14.84 & 14.77 29.61 epoch 15 avg 12
14.87 & 14.77 29.64 epoch 17 avg 15
14.88 & 14.77 29.65 epoch 20 avg 19
14.86 & 14.8 29.66 epoch 12 avg 8
14.86 & 14.81 29.67 epoch 14 avg 11
14.9 & 14.82 29.72 epoch 11 avg 6
14.9 & 14.82 29.72 epoch 16 avg 14
14.92 & 14.81 29.73 epoch 19 avg 18
14.9 & 14.84 29.74 epoch 13 avg 10
14.86 & 14.89 29.75 epoch 11 avg 2
14.91 & 14.86 29.77 epoch 15 avg 13
14.88 & 14.89 29.77 epoch 13 avg 1
14.94 & 14.84 29.78 epoch 18 avg 17
14.92 & 14.87 29.79 epoch 11 avg 7
14.89 & 14.9 29.79 epoch 12 avg 1
14.94 & 14.86 29.80 epoch 10 avg 4
14.97 & 14.85 29.82 epoch 10 avg 3
14.95 & 14.87 29.82 epoch 10 avg 5
14.96 & 14.9 29.86 epoch 12 avg 9
14.97 & 14.89 29.86 epoch 17 avg 16
14.96 & 14.91 29.87 epoch 14 avg 12
14.98 & 14.93 29.91 epoch 10 avg 2
15.01 & 14.91 29.92 epoch 10 avg 6
15.01 & 14.93 29.94 epoch 9 avg 3
15.01 & 14.93 29.94 epoch 16 avg 15
15.03 & 14.94 29.97 epoch 9 avg 4
15.02 & 14.95 29.97 epoch 13 avg 11
15.03 & 14.95 29.98 epoch 11 avg 8
15.03 & 14.99 30.02 epoch 15 avg 14
15.1 & 14.94 30.04 epoch 9 avg 2
15.02 & 15.02 30.04 epoch 11 avg 1
15.08 & 14.98 30.06 epoch 9 avg 5
15.07 & 14.99 30.06 epoch 12 avg 10
15.1 & 15.0 30.10 epoch 10 avg 7
15.1 & 15.04 30.14 epoch 14 avg 13
15.11 & 15.04 30.15 epoch 8 avg 3
15.12 & 15.05 30.17 epoch 9 avg 6
15.14 & 15.07 30.21 epoch 11 avg 9
15.15 & 15.1 30.25 epoch 8 avg 2
15.1 & 15.15 30.25 epoch 10 avg 1
15.17 & 15.08 30.25 epoch 13 avg 12
15.17 & 15.09 30.26 epoch 8 avg 4
15.17 & 15.1 30.27 epoch 10 avg 8
15.19 & 15.14 30.33 epoch 9 avg 1
15.21 & 15.13 30.34 epoch 12 avg 11
15.23 & 15.12 30.35 epoch 8 avg 5
15.24 & 15.13 30.37 epoch 9 avg 7
15.25 & 15.19 30.44 epoch 7 avg 3
15.23 & 15.22 30.45 epoch 7 avg 4
15.28 & 15.2 30.48 epoch 11 avg 10
15.24 & 15.24 30.48 epoch 7 avg 2
15.31 & 15.22 30.53 epoch 8 avg 6
15.31 & 15.23 30.54 epoch 10 avg 9
15.34 & 15.25 30.59 epoch 8 avg 1
15.33 & 15.31 30.64 epoch 7 avg 5
15.37 & 15.29 30.66 epoch 9 avg 8
15.34 & 15.33 30.67 epoch 6 avg 2
15.34 & 15.33 30.67 epoch 6 avg 3
15.41 & 15.39 30.80 epoch 6 avg 4
15.45 & 15.36 30.81 epoch 8 avg 7
15.45 & 15.38 30.83 epoch 7 avg 1
15.54 & 15.44 30.98 epoch 7 avg 6
15.5 & 15.49 30.99 epoch 5 avg 2
15.51 & 15.49 31.00 epoch 6 avg 1
15.56 & 15.54 31.10 epoch 5 avg 3
15.62 & 15.54 31.16 epoch 6 avg 5
15.61 & 15.65 31.26 epoch 5 avg 1
15.76 & 15.66 31.42 epoch 5 avg 4
15.73 & 15.72 31.45 epoch 4 avg 2
15.75 & 15.84 31.59 epoch 4 avg 1
15.93 & 15.83 31.76 epoch 4 avg 3
16.08 & 16.1 32.18 epoch 3 avg 1
16.2 & 16.06 32.26 epoch 3 avg 2
16.59 & 16.45 33.04 epoch 2 avg 1

Ablation Study

Skip warm-up

dev & test sum config
14.27 & 14.22 28.49 epoch 20 avg 5
14.28 & 14.23 28.51 epoch 20 avg 6
14.29 & 14.25 28.54 epoch 20 avg 8
14.33 & 14.23 28.56 epoch 20 avg 7
14.33 & 14.27 28.60 epoch 20 avg 4
14.33 & 14.28 28.61 epoch 20 avg 9
14.36 & 14.27 28.63 epoch 20 avg 3
14.33 & 14.3 28.63 epoch 19 avg 6
14.33 & 14.31 28.64 epoch 19 avg 5
14.34 & 14.31 28.65 epoch 19 avg 7
14.35 & 14.32 28.67 epoch 19 avg 4
14.36 & 14.31 28.67 epoch 20 avg 10
14.37 & 14.32 28.69 epoch 20 avg 2
14.39 & 14.31 28.70 epoch 18 avg 4
14.38 & 14.32 28.70 epoch 20 avg 11
14.39 & 14.32 28.71 epoch 19 avg 8
14.41 & 14.31 28.72 epoch 18 avg 5
14.4 & 14.32 28.72 epoch 18 avg 6
14.39 & 14.34 28.73 epoch 18 avg 3
14.41 & 14.34 28.75 epoch 19 avg 9
14.43 & 14.33 28.76 epoch 18 avg 7
14.41 & 14.36 28.77 epoch 19 avg 3
14.44 & 14.36 28.80 epoch 18 avg 8
14.45 & 14.35 28.80 epoch 19 avg 10
14.44 & 14.36 28.80 epoch 20 avg 12
14.44 & 14.37 28.81 epoch 17 avg 5
14.46 & 14.38 28.84 epoch 17 avg 7
14.46 & 14.38 28.84 epoch 20 avg 13
14.46 & 14.39 28.85 epoch 17 avg 6
14.48 & 14.37 28.85 epoch 18 avg 9
14.47 & 14.38 28.85 epoch 19 avg 11
14.49 & 14.37 28.86 epoch 17 avg 4
14.42 & 14.45 28.87 epoch 18 avg 2
14.5 & 14.38 28.88 epoch 17 avg 3
14.5 & 14.4 28.90 epoch 18 avg 10
14.49 & 14.42 28.91 epoch 17 avg 8
14.48 & 14.43 28.91 epoch 19 avg 12
14.5 & 14.43 28.93 epoch 16 avg 4
14.51 & 14.42 28.93 epoch 16 avg 5
14.53 & 14.4 28.93 epoch 16 avg 6
14.48 & 14.45 28.93 epoch 20 avg 1
14.5 & 14.44 28.94 epoch 20 avg 14
14.5 & 14.45 28.95 epoch 17 avg 9
14.5 & 14.47 28.97 epoch 17 avg 2
14.55 & 14.44 28.99 epoch 16 avg 3
14.53 & 14.46 28.99 epoch 18 avg 11
14.54 & 14.47 29.01 epoch 16 avg 7
14.55 & 14.48 29.03 epoch 19 avg 13
14.56 & 14.48 29.04 epoch 15 avg 5
14.6 & 14.45 29.05 epoch 19 avg 2
14.57 & 14.48 29.05 epoch 16 avg 8
14.56 & 14.49 29.05 epoch 20 avg 15
14.58 & 14.49 29.07 epoch 17 avg 10
14.57 & 14.51 29.08 epoch 15 avg 4
14.58 & 14.5 29.08 epoch 16 avg 2
14.57 & 14.51 29.08 epoch 18 avg 12
14.59 & 14.52 29.11 epoch 19 avg 14
14.6 & 14.54 29.14 epoch 15 avg 3
14.62 & 14.52 29.14 epoch 15 avg 6
14.63 & 14.51 29.14 epoch 16 avg 9
14.62 & 14.53 29.15 epoch 17 avg 11
14.62 & 14.55 29.17 epoch 14 avg 4
14.62 & 14.55 29.17 epoch 18 avg 13
14.63 & 14.55 29.18 epoch 20 avg 16
14.61 & 14.59 29.20 epoch 14 avg 3
14.67 & 14.53 29.20 epoch 15 avg 7
14.65 & 14.56 29.21 epoch 14 avg 5
14.67 & 14.54 29.21 epoch 15 avg 8
14.65 & 14.56 29.21 epoch 16 avg 10
14.65 & 14.59 29.24 epoch 19 avg 15
14.66 & 14.59 29.25 epoch 17 avg 12
14.64 & 14.61 29.25 epoch 18 avg 1
14.69 & 14.58 29.27 epoch 14 avg 6
14.68 & 14.59 29.27 epoch 14 avg 7
14.68 & 14.59 29.27 epoch 15 avg 9
14.66 & 14.63 29.29 epoch 17 avg 1
14.7 & 14.61 29.31 epoch 16 avg 11
14.67 & 14.64 29.31 epoch 14 avg 2
14.61 & 14.71 29.32 epoch 16 avg 1
14.7 & 14.62 29.32 epoch 18 avg 14
14.7 & 14.63 29.33 epoch 19 avg 1
14.7 & 14.65 29.35 epoch 13 avg 3
14.72 & 14.63 29.35 epoch 13 avg 4
14.73 & 14.62 29.35 epoch 13 avg 6
14.72 & 14.63 29.35 epoch 20 avg 17
14.73 & 14.63 29.36 epoch 13 avg 5
14.72 & 14.64 29.36 epoch 14 avg 8
14.73 & 14.63 29.36 epoch 15 avg 2
14.71 & 14.66 29.37 epoch 17 avg 13
14.76 & 14.62 29.38 epoch 15 avg 10
14.74 & 14.68 29.42 epoch 19 avg 16
14.76 & 14.68 29.44 epoch 16 avg 12
14.78 & 14.68 29.46 epoch 14 avg 9
14.78 & 14.69 29.47 epoch 13 avg 2
14.76 & 14.71 29.47 epoch 18 avg 15
14.8 & 14.69 29.49 epoch 12 avg 4
14.81 & 14.68 29.49 epoch 13 avg 7
14.78 & 14.72 29.50 epoch 12 avg 5
14.77 & 14.74 29.51 epoch 17 avg 14
14.82 & 14.7 29.52 epoch 15 avg 11
14.82 & 14.71 29.53 epoch 12 avg 3
14.8 & 14.73 29.53 epoch 20 avg 18
14.79 & 14.76 29.55 epoch 12 avg 2
14.84 & 14.71 29.55 epoch 13 avg 8
14.81 & 14.76 29.57 epoch 16 avg 13
14.84 & 14.75 29.59 epoch 12 avg 6
14.85 & 14.74 29.59 epoch 14 avg 10
14.84 & 14.75 29.59 epoch 15 avg 1
14.82 & 14.77 29.59 epoch 19 avg 17
14.85 & 14.79 29.64 epoch 15 avg 12
14.85 & 14.8 29.65 epoch 18 avg 16
14.87 & 14.79 29.66 epoch 11 avg 4
14.88 & 14.79 29.67 epoch 13 avg 9
14.9 & 14.78 29.68 epoch 11 avg 5
14.9 & 14.78 29.68 epoch 12 avg 7
14.91 & 14.78 29.69 epoch 11 avg 3
14.84 & 14.85 29.69 epoch 14 avg 1
14.87 & 14.84 29.71 epoch 11 avg 2
14.89 & 14.83 29.72 epoch 17 avg 15
14.9 & 14.83 29.73 epoch 14 avg 11
14.91 & 14.83 29.74 epoch 20 avg 19
14.89 & 14.89 29.78 epoch 13 avg 1
14.92 & 14.86 29.78 epoch 16 avg 14
14.92 & 14.88 29.80 epoch 19 avg 18
14.98 & 14.83 29.81 epoch 11 avg 6
14.96 & 14.85 29.81 epoch 12 avg 8
14.96 & 14.87 29.83 epoch 10 avg 4
14.98 & 14.87 29.85 epoch 13 avg 10
14.96 & 14.91 29.87 epoch 18 avg 17
15.0 & 14.88 29.88 epoch 10 avg 3
14.99 & 14.89 29.88 epoch 15 avg 13
14.98 & 14.91 29.89 epoch 12 avg 1
15.03 & 14.9 29.93 epoch 10 avg 5
15.0 & 14.94 29.94 epoch 14 avg 12
15.01 & 14.93 29.94 epoch 17 avg 16
15.02 & 14.93 29.95 epoch 11 avg 7
15.08 & 14.92 30.00 epoch 10 avg 2
15.08 & 14.93 30.01 epoch 12 avg 9
15.06 & 14.96 30.02 epoch 9 avg 3
15.08 & 14.98 30.06 epoch 16 avg 15
15.09 & 14.98 30.07 epoch 13 avg 11
15.06 & 15.03 30.09 epoch 9 avg 2
15.11 & 14.98 30.09 epoch 10 avg 6
15.11 & 14.99 30.10 epoch 9 avg 4
15.08 & 15.04 30.12 epoch 11 avg 1
15.14 & 15.0 30.14 epoch 11 avg 8
15.13 & 15.02 30.15 epoch 15 avg 14
15.16 & 15.05 30.21 epoch 12 avg 10
15.16 & 15.06 30.22 epoch 14 avg 13
15.18 & 15.05 30.23 epoch 9 avg 5
15.18 & 15.08 30.26 epoch 8 avg 3
15.21 & 15.05 30.26 epoch 10 avg 7
15.22 & 15.1 30.32 epoch 11 avg 9
15.23 & 15.11 30.34 epoch 10 avg 1
15.22 & 15.12 30.34 epoch 13 avg 12
15.21 & 15.15 30.36 epoch 8 avg 2
15.22 & 15.16 30.38 epoch 9 avg 1
15.27 & 15.14 30.41 epoch 9 avg 6
15.29 & 15.14 30.43 epoch 8 avg 4
15.3 & 15.15 30.45 epoch 10 avg 8
15.31 & 15.19 30.50 epoch 12 avg 11
15.38 & 15.22 30.60 epoch 8 avg 5
15.37 & 15.23 30.60 epoch 11 avg 10
15.37 & 15.24 30.61 epoch 9 avg 7
15.4 & 15.23 30.63 epoch 7 avg 2
15.35 & 15.3 30.65 epoch 8 avg 1
15.43 & 15.24 30.67 epoch 7 avg 3
15.45 & 15.29 30.74 epoch 10 avg 9
15.48 & 15.29 30.77 epoch 7 avg 4
15.46 & 15.33 30.79 epoch 8 avg 6
15.46 & 15.39 30.85 epoch 7 avg 1
15.52 & 15.35 30.87 epoch 9 avg 8
15.57 & 15.39 30.96 epoch 6 avg 2
15.58 & 15.38 30.96 epoch 7 avg 5
15.63 & 15.41 31.04 epoch 6 avg 3
15.63 & 15.44 31.07 epoch 8 avg 7
15.61 & 15.48 31.09 epoch 6 avg 1
15.68 & 15.43 31.11 epoch 6 avg 4
15.72 & 15.49 31.21 epoch 7 avg 6
15.76 & 15.53 31.29 epoch 5 avg 2
15.8 & 15.55 31.35 epoch 5 avg 3
15.81 & 15.55 31.36 epoch 6 avg 5
15.82 & 15.66 31.48 epoch 5 avg 1
15.9 & 15.68 31.58 epoch 5 avg 4
15.91 & 15.73 31.64 epoch 4 avg 2
16.02 & 15.82 31.84 epoch 4 avg 1
16.02 & 15.83 31.85 epoch 4 avg 3
16.16 & 16.05 32.21 epoch 3 avg 1
16.18 & 16.05 32.23 epoch 3 avg 2
16.54 & 16.47 33.01 epoch 2 avg 1

@yfyeung yfyeung changed the title Add mux for finetune Add adapting recipe for pruned_transducer_stateless7 May 15, 2023
@yfyeung yfyeung changed the title Add adapting recipe for pruned_transducer_stateless7 Add adaption recipe for pruned_transducer_stateless7 May 15, 2023
marcoyang1998
marcoyang1998 reviewed May 15, 2023
View reviewed changes
egs/librispeech/ASR/pruned_transducer_stateless7/finetune.py Outdated
@@ -104,6 +103,7 @@ def set_batch_count(model: Union[nn.Module, DDP], batch_count: float) -> None:

def add_finetune_arguments(parser: argparse.ArgumentParser):
parser.add_argument("--do-finetune", type=str2bool, default=False)
parser.add_argument("--use-mux", type=str2bool, default=False)
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you please add some documentation for this argument?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure

@danpovey
Copy link
Collaborator

We should probably skip warmup even though it doesn't seem to help results, because it's a bit dangerous to include warmup code in an already-trained model (could cause regression if learning rate is too high).

@yfyeung
Copy link
Collaborator Author

yfyeung commented May 17, 2023 •
edited
Loading

We should probably skip warmup even though it doesn't seem to help results, because it's a bit dangerous to include warmup code in an already-trained model (could cause regression if learning rate is too high).

I see. I will merge the skip-warmup version later.

@yfyeung yfyeung merged commit 562bda9 into k2-fsa:master May 17, 2023
@yfyeung yfyeung deleted the mux branch May 17, 2023 08:02
@yfyeung yfyeung changed the title Add adaption recipe for pruned_transducer_stateless7 Add mux to fine-tuning recipe for pruned_transducer_stateless7 May 20, 2023
@yfyeung yfyeung changed the title Add mux to fine-tuning recipe for pruned_transducer_stateless7 Add mux to fine-tune recipe for pruned_transducer_stateless7 May 20, 2023
@yfyeung yfyeung mentioned this pull request May 20, 2023
Merged
@yfyeung
Copy link
Collaborator Author

yfyeung commented May 20, 2023

Docs: #1074

@HsunGong
Copy link

@yfyeung hello, could you please provide the config of Fine-tune from LibriSpeech 13.59 & 13.49 | epoch 11 avg 10, I use the same command CMD above without mux and only get 14.66 with epoch 11 avg 10.

CMD: ./pruned_transducer_stateless7/finetune.py \ --world-size 2 \ --num-epochs 20 \ --start-epoch 1 \ --exp-dir pruned_transducer_stateless7/exp_giga_finetune \ --subset S \ --use-fp16 1 \ --base-lr 0.005 \ --lr-epochs 100 \ --lr-batches 100000 \ --bpe-model icefall-asr-librispeech-pruned-transducer-stateless7-2022-11-11/data/lang_bpe_500/bpe.model \ --do-finetune True \ --use-mux True \ --finetune-ckpt icefall-asr-librispeech-pruned-transducer-stateless7-2022-11-11/exp/pretrained.pt \ --max-duration 500

There is only one difference, I use pruned_transducer_stateless7_ctc as the pretrained model (without ctc-loss-scale during finetuning).

Moreover, why epoch 11 avg 5 is worse than epoch 11 avg 10 while the valid loss is getting smaller? Do you know?

THX

@marcoyang1998
Copy link
Collaborator

could you please provide the config of Fine-tune from LibriSpeech 13.59 & 13.49 | epoch 11 avg 10

Please have a look at #944.

@HsunGong HsunGong mentioned this pull request Sep 1, 2023
Merged
@marcoyang1998 marcoyang1998 mentioned this pull request Oct 17, 2023
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@marcoyang1998 marcoyang1998 marcoyang1998 left review comments

@csukuangfj csukuangfj csukuangfj approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@yfyeung @danpovey @HsunGong @marcoyang1998 @csukuangfj