Kohya_ss GUI PURE edition

[bmaltais]

Starting a discussion about work being done to strip all sd-scripts code from this repo and use a "cloned" version of kohya_ss sd-scripts as the true source of python code to execute the training.

Since this is a major refactor of most of the code I think it is worth discussion issues you might encounter with it. The current refactored code is in the dev-pure branch.

Pleade give it a try by checking out that branch with:

git pull
git checkout dev-pure

Then start the gui as usual... hope everything will fall in place properly... this is the goal... but you will notice a much lighter file structure in the repo as now everything is referenced from the sd-scripts folder that get cloned at setup / run time.

[bmaltais]

I did run a test under WLS2 on the PURE codebse and all look good. Both Windows and Linux (Ubuntu) appear pretty solid.

[311-code]

Sounds pretty great idea to me, what do you think the benefits of doing this potentially be? (I have no idea) I will help test this.

[bmaltais]

From a user point view it is fairly invisible, but this open the door for easier maintenance of the GUI.

[bmaltais]

@jim60105 OK, let's discuss the git submodule vs .gitignore local clone approach. I am really unsure of which is the least risky and least trouble to do upgrade and move to other branch for users. With the local clone method I know I could easilly tell a user to:

git pull
git checkout <branch name>
.\gui.bat or ./gui.sh

And the sd-scripts folder will be transparently updated... But I am unsure what the actual impact will be if I use submodule. My past experience make me think it will be much more troublesome for users to do something similar...

For you Dockerfile you could technically use the git submodule road as part of your build... especially if I specify a commit hash instead of a tag in the .sd-scripts-release file...

What do you think?

[jim60105]

git pull
git checkout <branch name>
git submodule update --init --recursive
.\gui.bat or ./gui.sh

Notice that the git submodule command only need to execute once, after that submodules will be checkout when executing git checkout in the main repository.
I have updated my concept, update is needed.

In terms of Dockerfile I implemented, all the project files should be obtained before docker building.
That is, if the user do git clone --recursive, the Dockerfile will work as usual.

Generally speaking, I do not perform git clone in Dockerfile.
I rarely do this, except in some CI/CD scenarios where I need to automate the building of specific versions of projects. Often doing this is where the Dockerfile becomes more and more complicated.

[jim60105]

In addition, I personally prefer to use git fetch instead of git pull, because git pull implies merge, which may create unexpected commits.

[bmaltais]

I have updated the dev-pure branch to use a submodule. It is as much trouble to maintain as git clone but if it is more standard and easier for other devs then why not ;-) Look at the last commit and let me know if you find an issue with it.

I am using this function as part of setup and validation for both Linux and Windows:

def update_submodule(submodule_path, branch_or_tag):
    """
    Ensure the submodule is initialized, updated, and set to a specific commit, tag, or branch.
    
    Parameters:
    - submodule_path: The relative path within the repository to the submodule.
    - branch_or_tag: The specific commit, tag, or branch to checkout in the submodule.
    """
    original_dir = os.getcwd()  # Store the original directory
    try:
        # Ensure the working directory is the root of the main repository
        if not os.path.exists(submodule_path):
            raise FileNotFoundError(f"Submodule path does not exist: {submodule_path}")
        
        # Initialize and update the submodule
        subprocess.run(["git", "submodule", "update", "--init", "--recursive", "--quiet", submodule_path], check=True)
        log.info("Submodule initialized and updated.")
        
        # Navigate to the submodule directory
        os.chdir(submodule_path)
        
        # Fetch the latest changes from the remote, including tags
        subprocess.run(["git", "fetch", "--all", "--tags", "--quiet"], check=True)
        
        # Checkout the specified branch, tag, or commit
        subprocess.run(["git", "checkout", "--quiet", branch_or_tag], check=True)
        log.info(f"Submodule set to {branch_or_tag}.")
        
    except subprocess.CalledProcessError as e:
        log.error(f"Error during Git operation: {e}")
    except FileNotFoundError as e:
        log.error(e)
    finally:
        os.chdir(original_dir)  # Restore the original directory

I will specify the sd-scripts tag or commit in the .sd-scripts-release file in the root of my repo... so you can pick it up for the docker build to make sure the sd-script folder point to the appropriate sd-scripts release for the GUI

[jim60105]

git submodule is fixed on commit, not branch or tags
Unless you have such a need, just do git checkout in the parent folder and then git submodule update --init --recursive all the submodules will update. There is no need to go into the submodule folder for git operations.

[jim60105]

Other than that, I haven't tried docker build yet (it takes a little time to build such a large image), but I think it looks good

[bmaltais]

Is it possible to change the submodule commit without going in the sub module folder? Or is it possible to specify it in the .gitmodules file?

I need to ensure users get the as-scripts specific release for the gui release… If that can be done without going in the sd-scripts folder than bonus.

[bmaltais]

Ohhhh... if I git checkout in the sd-scripts it will actually get commited to the repo and this lock that version so that when a git submodule update --init --recursive is done that specific version will be used... brilliant ;-)

[jim60105]

git checkout --recurse-submodules should also do the trick during checkout.
I'm not sure if it does --init though, need to check the documentation.

[bmaltais]

Commited updated code. Much simpled this way. I have not used submodules much so learning as I go ;-)

def update_submodule():
    """
    Ensure the submodule is initialized and updated.
    """
    try:
        # Initialize and update the submodule
        subprocess.run(["git", "submodule", "update", "--init", "--recursive", "--quiet"], check=True)
        log.info("Submodule initialized and updated.")
        
    except subprocess.CalledProcessError as e:
        log.error(f"Error during Git operation: {e}")
    except FileNotFoundError as e:
        log.error(e)

[jim60105]

I have submit a PR for this changes #2032

[bmaltais]

Excellent, merged into dev-pure. This is looking pretty solid.

[bmaltais]

@wkpark If you have a functional a1111 extension I could add something about it in the README. Le me know and let's add something if it is the case.

[bmaltais]

I am updating the setup for windows to make things better. The old bitsandbytes files that were copied are causing more issues than good. I have disabled that from the setup code. Instead, installing the latest https://github.com/jllllll/bitsandbytes-windows-webui/releases/download/wheels/bitsandbytes-0.41.2.post2-py3-none-win_amd64.whl work just as well.

I have also discovered a python module for cudnn that can install the required updated cudnn files without having to download them manually on windows. I have updated the instructions to suggest installing those for users with 39x0 and 40x0 cards.

This will turn out to be some major release. @wkpark has started work to make the GUI more streamlined (interestingly going back to the old accordions style rather than tabs to keep everything on the same page)... and a bunch of other things. Keep an eye in the Pull Request section if you want to discuss/contribute.

He is creating an extension for a1111 for the GUI and all this rework we are doing will make it possible to train directly in user's prefered SD UI.

[bmaltais]

@wkpark Regarding the list offered to the users, I added an empty choice at the begining of each list to make sure it was not causing confusion to the users and issues with the feilds that are optional. You may have a better solution but for now this was a quick and dirty fix.