May version finetuning degrades models capabilities. April version works nice.

[kgonia]

So I have April version on my local machine and on AWS AMI

commit 1e645deebec16ea5c3740ed0903ebbe21f273e1b
Merge: af0df34 cd8f85d
Author: bmaltais <[email protected]>
Date:   Sat Apr 1 16:52:37 2023 -0400

    Merge pull request #512 from bmaltais/dev

    v21.4.0

with this version I'm able to finetune with specific topic without degrading model capabilities.

Recently I updated code base to newer version

 commit 959ef18e2dec8cbdefa3571bbb84cd0d01ff9890 (origin/master, origin/HEAD)
Merge: fa41e40 eb2ca44
Author: bmaltais <[email protected]>
Date:   Wed May 3 13:22:06 2023 -0400

    Merge pull request #706 from chiragjn/patch-1

    Add `--no-cache-dir` to reduce image size

and it's degrades model capability.

I'm using same data and same configuration but got totally different result.

Below example of test generation from same prompt during training after first epoch

new code base

old code base

Was there any changes that can affect finetuning?

[bmaltais]

Indeed Kohya constantly update his code... Probably a change that is having a negative side effect. Really hard to pinpoint what it might be....

[rafstahelin]

So I have April version on my local machine and on AWS AMI

commit 1e645deebec16ea5c3740ed0903ebbe21f273e1b

Merge: af0df34 cd8f85d

Author: bmaltais <[email protected]>

Date:   Sat Apr 1 16:52:37 2023 -0400



    Merge pull request #512 from bmaltais/dev



    v21.4.0

with this version I'm able to finetune with specific topic without degrading model capabilities.

Recently I updated code base to newer version

 commit 959ef18e2dec8cbdefa3571bbb84cd0d01ff9890 (origin/master, origin/HEAD)

Merge: fa41e40 eb2ca44

Author: bmaltais <[email protected]>

Date:   Wed May 3 13:22:06 2023 -0400



    Merge pull request #706 from chiragjn/patch-1



    Add `--no-cache-dir` to reduce image size

and it's degrades model capability.

I'm using same data and same configuration but got totally different result.

Below example of test generation from same prompt during training after first epoch

new code base

old code base

Was there any changes that can affect finetuning?

My Kohya can't even fine tune. Lora is fine, but get error message when I start fine tune message.
Should I be on 3.10.6 Python rather than 3.10.9?

[kgonia]

I'm using 3.10.10 curently