Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add option --sample_at_first #907

Merged
merged 3 commits into from
Dec 3, 2023

Conversation

shirayu
Copy link
Contributor

@shirayu shirayu commented Oct 29, 2023

  • ✨ Generate samples before training
    • This is useful for comparison before and during the training

🗒️ Note

There is a possibility that the images generated by this and the images generated by sdxl_gen_img.py (or other generation scripts) are slightly different, even with the same parameters. Whether this is true or not has not yet been investigated.

@shirayu
Copy link
Contributor Author

shirayu commented Oct 30, 2023

I have noticed that the reason for the difference in results may be due to --full_bf16 or --full_bf16.
It would be better to add is_full argument to sdxl_train_util.load_target_model and convert the unet, text_encoder1, text_encoder2 in it.

Related to #628

@araleza
Copy link

araleza commented Nov 1, 2023

This seems like a good feature. I found myself wanting it just last week.

@kohya-ss kohya-ss merged commit 383b4a2 into kohya-ss:dev Dec 3, 2023
1 check passed
@kohya-ss
Copy link
Owner

kohya-ss commented Dec 3, 2023

Thank you for this. I think the difference in results may be acceptable.

In addition, I have another issue in generating samples during training (I got strange sepia images), so I cannot confirm the difference yet.

@kohya-ss
Copy link
Owner

kohya-ss commented Dec 3, 2023

I found that if --sample_every_n_epochs is set, the script generates the images two times every N epochs after merging. I think this is because the following lines are in the training loop (for epoch ...).

        # For --sample_at_first
        self.sample_images(accelerator, args, epoch, global_step, accelerator.device, vae, tokenizer, text_encoder, unet)

To solve this, I will move these two lines just before the loop.

@shirayu shirayu deleted the add_option_sample_at_first branch December 3, 2023 14:35
kohya-ss added a commit that referenced this pull request Dec 7, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants