Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fix DAT models download #16302

Merged
merged 5 commits into from
Oct 24, 2024
Merged

Fix DAT models download #16302

merged 5 commits into from
Oct 24, 2024

Conversation

w-e-w
Copy link
Collaborator

@w-e-w w-e-w commented Jul 31, 2024 •
edited
Loading

Description

Fix DAT models download

the the download links we used was uploaded by n0kovo because the orignal due to the orignal was hosted on Google Drive / Baidu Disk, the issue is n0kovo used LFS,
LFS on GitHub quota is exceeded for obvious reasons and so the Auto download doesn't work

Worse it currently downloads the LFS pointer file so webui thinks that thinks is downloaded and errors

TODO

not sure if it will be in this PR or a separate one

  • loading local upscaler files seems to be a bit disorganized
    it seems like every architecture just seem to implement its own file loader (not model loader file loader)
    this seems to result in various bugs
    for DAT you will see duplicate entries if local files exist
    some other upscallers don't seem to list custom user models

Checklist:

@w-e-w
UpscalerData add sha256
2b2302b
@w-e-w
download DAT models from huggingface
2f74cd6 
the LFS from GitHub "is over its data quota"
@w-e-w w-e-w marked this pull request as ready for review July 31, 2024 10:11
@w-e-w w-e-w changed the title Fix dat models download Fix DAT models download Jul 31, 2024
@n0kovo
Copy link
Contributor

n0kovo commented Aug 10, 2024 •
edited
Loading

Well that explains a lot. Didn't realize there was a quota, but of course there is.

@w-e-w
Copy link
Collaborator Author

w-e-w commented Aug 10, 2024 •
edited
Loading

I guess that your Git LFS Data on GitHub Billing page has exploded

what sort of building plan are you on, did you get charged for the LFS data?

@w-e-w
Copy link
Collaborator Author

w-e-w commented Oct 17, 2024 •
edited
Loading

new commit rewrite load_file_from_url with requests, move to util

been getting this error when using torch.hub import download_url_to_file.download_url_to_file to download files from hugging face

urllib.error.URLError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:997)>

but request seems to work
so I rewritten load_file_from_url to use request

I also move it modules.modelloader.load_file_from_url to modules.utils.load_file_from_url
the reason is that it's a general useful function that someone may wish to use elsewhere

@w-e-w w-e-w mentioned this pull request Oct 17, 2024
Open
4 tasks
@w-e-w w-e-w changed the title Fix DAT models download [Fix]: DAT models download Oct 19, 2024
@w-e-w w-e-w changed the title [Fix]: DAT models download Fix DAT models download Oct 19, 2024
@catboxanon catboxanon merged commit 984b952 into dev Oct 24, 2024
6 checks passed
@catboxanon catboxanon deleted the Fix-DAT-models-download branch October 24, 2024 13:05
bluelovers added a commit to bluelovers/stable-diffusion-webui that referenced this pull request Oct 25, 2024
@bluelovers
Merge remote-tracking branch 'remotes/bluelovers/patch-3' into dev-lo…
29d038a 
…cal-202406

* remotes/bluelovers/patch-3:
  chore(js): avoid lots of `Wake Lock is not supported.`
  Fix DAT models download (AUTOMATIC1111#16302)
@w-e-w w-e-w mentioned this pull request Oct 27, 2024
Closed
4 tasks
kafffka-cat added a commit to kafffka-cat/stable-diffusion-webui that referenced this pull request Oct 30, 2024
@kafffka-cat
Squashed commit of the following:
d3cb8fd 
commit d712b4f40a05498bc80ae83c8ea7ffaad6c9ece1
Author: kafffka-cat <[email protected]>
Date:   Wed Oct 30 20:15:56 2024 +0300

    Update sd_models.py

commit 02fca85de807ece7b8caeb379ba1c7b38b50339c
Merge: 3b3bc10b e76d0a2
Author: kafffka-cat <[email protected]>
Date:   Wed Oct 30 18:55:01 2024 +0300

    Merge branch 'master' of https://github.com/Kittensx/stable-diffusion-webui into pr/16598

commit 3b3bc10bbb35ec3e3bd69a1548fc9bf9364a1afe
Author: kafffka-cat <[email protected]>
Date:   Wed Oct 30 18:53:54 2024 +0300

    Squashed commit of the following:

    commit acf8a5b6651f1916df81e0010dae9428bc8e002a
    Merge: 0d874a6 7a91722
    Author: kafffka-cat <[email protected]>
    Date:   Wed Oct 30 18:43:10 2024 +0300

        Merge branch 'pr/16594' into pr/16290

    commit 7a91722
    Author: Won-Kyu Park <[email protected]>
    Date:   Sat Oct 26 23:32:29 2024 +0900

        support loading libtcmalloc_minimal.dll for windows

    commit 5865da2
    Merge: bb1f391 c2ce1d3
    Author: w-e-w <[email protected]>
    Date:   Sun Oct 20 11:40:02 2024 +0900

        Merge pull request AUTOMATIC1111#16569 from AUTOMATIC1111/feat/ztsnr-auto

        Automatically enable ztSNR based on existence of key in `state_dict`

    commit bb1f391
    Author: w-e-w <[email protected]>
    Date:   Sun Oct 20 09:58:53 2024 +0900

        clarify readme: weget ... chmod +x webui.sh (AUTOMATIC1111#16251)

    commit 6a59766
    Author: w-e-w <[email protected]>
    Date:   Sun Oct 20 09:56:12 2024 +0900

        Add Skip Early CFG to XYZ (AUTOMATIC1111#16282)

        Co-authored-by: Yevhenii Hurin <[email protected]>

    commit 65423d2
    Author: w-e-w <[email protected]>
    Date:   Sun Oct 20 09:52:47 2024 +0900

        MIME type text/css (AUTOMATIC1111#16406)

    commit c2bc187
    Author: w-e-w <[email protected]>
    Date:   Sun Oct 20 09:51:59 2024 +0900

        fix modalImageViewer preview/result flicker (AUTOMATIC1111#16426)

    commit d0b27dc
    Merge: bb4cbaf cbaaf0a
    Author: w-e-w <[email protected]>
    Date:   Sun Oct 20 09:42:21 2024 +0900

        Merge pull request AUTOMATIC1111#16300 from hello2564/fix_NGMS_pr_typo

        fix NGMS pr typo

    commit c2ce1d3
    Author: catboxanon <[email protected]>
    Date:   Sat Oct 19 19:58:13 2024 -0400

        Automatically enable ztSNR based on existence of key in state_dict

    commit bb4cbaf
    Merge: c462e5a 9677b09
    Author: w-e-w <[email protected]>
    Date:   Sun Oct 20 08:27:12 2024 +0900

        Merge pull request AUTOMATIC1111#16341 from gutris1/devv

        add break-word for geninfo in pnginfo

    commit c462e5a
    Merge: 8b19b75 c9a06d1
    Author: catboxanon <[email protected]>
    Date:   Sat Oct 19 10:51:16 2024 -0400

        Merge pull request AUTOMATIC1111#16460 from AUTOMATIC1111/sd-1.5-url

    commit 8b19b75
    Merge: 907bfb5 1ae073c
    Author: AUTOMATIC1111 <[email protected]>
    Date:   Sat Oct 19 17:40:56 2024 +0300

        Merge pull request AUTOMATIC1111#16567 from AUTOMATIC1111/feat/sdxl-vpred

        Support and automatically detect SDXL V-prediction models

    commit 907bfb5
    Author: AUTOMATIC1111 <[email protected]>
    Date:   Sat Oct 19 17:33:58 2024 +0300

        add w-e-w and catboxanon to codeowners file

    commit 1ae073c
    Author: catboxanon <[email protected]>
    Date:   Sat Oct 19 06:53:19 2024 -0400

        Support SDXL v-pred models

    commit c9a06d1
    Author: missionfloyd <[email protected]>
    Date:   Tue Oct 8 16:50:39 2024 -0600

        Use stable-diffusion-v1-5 repo instead

    commit f57ec2b
    Author: missionfloyd <[email protected]>
    Date:   Tue Sep 3 19:58:29 2024 -0600

        Update stable diffusion 1.5 URL

    commit 9677b09
    Author: gutris1 <[email protected]>
    Date:   Wed Aug 7 17:37:23 2024 +0700

        add break-word for geninfo in pnginfo

    commit cbaaf0a
    Author: hello2564 <[email protected]>
    Date:   Wed Jul 31 14:55:30 2024 +0800

        fix NGMS pr typo

    commit 0d874a6
    Author: Abhishek Vakharia <[email protected]>
    Date:   Sun Jul 28 21:23:34 2024 -0700

        fix move2move tab not showing

    commit 4823909
    Merge: 850e149 82a973c
    Author: AUTOMATIC1111 <[email protected]>
    Date:   Sat Jul 27 15:50:26 2024 +0300

        Merge branch 'master' into dev

    commit 850e149
    Merge: 834297b 8e0881d
    Author: AUTOMATIC1111 <[email protected]>
    Date:   Sat Jul 27 15:47:49 2024 +0300

        Merge pull request AUTOMATIC1111#16275 from AUTOMATIC1111/fix-image-upscale-on-cpu

        fix image upscale on cpu

    commit 8e0881d
    Author: w-e-w <[email protected]>
    Date:   Sat Jul 27 21:10:20 2024 +0900

        fix image upscale on cpu

        for some reason upscale using cpu will fail with
        RuntimeError: Inplace update to inference tensor outside InferenceMode
        switch from no_grad to inference_mode seems to have fixed it

    commit 834297b
    Merge: 9f5a98d c19d044
    Author: AUTOMATIC1111 <[email protected]>
    Date:   Sat Jul 27 07:09:08 2024 +0300

        Merge branch 'master' into dev

commit e79134c3f7d3becce2b9ed5584faacc7179d4088
Author: kafffka-cat <[email protected]>
Date:   Wed Oct 30 18:50:53 2024 +0300

    Squashed commit of the following:

    commit 920a3a4
    Author: w-e-w <[email protected]>
    Date:   Wed Jul 31 02:43:13 2024 +0900

        SD3 Lora page filter - detection not implemented

    commit 4823909
    Merge: 850e149 82a973c
    Author: AUTOMATIC1111 <[email protected]>
    Date:   Sat Jul 27 15:50:26 2024 +0300

        Merge branch 'master' into dev

    commit 850e149
    Merge: 834297b 8e0881d
    Author: AUTOMATIC1111 <[email protected]>
    Date:   Sat Jul 27 15:47:49 2024 +0300

        Merge pull request AUTOMATIC1111#16275 from AUTOMATIC1111/fix-image-upscale-on-cpu

        fix image upscale on cpu

    commit 8e0881d
    Author: w-e-w <[email protected]>
    Date:   Sat Jul 27 21:10:20 2024 +0900

        fix image upscale on cpu

        for some reason upscale using cpu will fail with
        RuntimeError: Inplace update to inference tensor outside InferenceMode
        switch from no_grad to inference_mode seems to have fixed it

    commit 834297b
    Merge: 9f5a98d c19d044
    Author: AUTOMATIC1111 <[email protected]>
    Date:   Sat Jul 27 07:09:08 2024 +0300

        Merge branch 'master' into dev

commit 834ba48f88f0176725771dc9d65bea1cb4a7e8fa
Author: kafffka-cat <[email protected]>
Date:   Wed Oct 30 18:50:32 2024 +0300

    Squashed commit of the following:

    commit 09771cb4669da07ca12972ffd1b1cb0be3af4461
    Merge: 6796147 77c0c16
    Author: kafffka-cat <[email protected]>
    Date:   Wed Oct 30 18:44:20 2024 +0300

        Merge branch 'sd-1.5-url' into warning-WebUI-is-installed-under-a-dot-directory

    commit 77c0c16
    Merge: de44ee2 e76d0a2
    Author: kafffka-cat <[email protected]>
    Date:   Wed Oct 30 17:32:27 2024 +0300

        Merge branch 'pr/16598' into sd-1.5-url

    commit de44ee2
    Merge: c9a06d1 4595f9a
    Author: kafffka-cat <[email protected]>
    Date:   Wed Oct 30 17:32:07 2024 +0300

        Merge branch 'pr/16553' into sd-1.5-url

    commit 6796147
    Author: w-e-w <[email protected]>
    Date:   Wed Oct 30 15:44:14 2024 +0900

        add version check

    commit e76d0a2
    Merge: 4ea5403 2571d78
    Author: Kittensx <[email protected]>
    Date:   Sun Oct 27 12:07:45 2024 -0400

        Merge pull request AUTOMATIC1111#4 from Kittensx/Kittensx-patch-2

        Add files via upload

    commit 2571d78
    Author: Kittensx <[email protected]>
    Date:   Sun Oct 27 12:06:08 2024 -0400

        Add files via upload

        forgot to include this in my lasst commit

    commit 4ea5403
    Merge: 6ffb728 da2e709
    Author: Kittensx <[email protected]>
    Date:   Sun Oct 27 11:08:53 2024 -0400

        Merge pull request AUTOMATIC1111#3 from Kittensx/Kittensx-patch-Simple-KES

        Update simple_karras_exponential_scheduler.py

    commit da2e709
    Author: Kittensx <[email protected]>
    Date:   Sun Oct 27 11:08:06 2024 -0400

        Update simple_karras_exponential_scheduler.py

    commit 6ffb728
    Merge: 82a973c 78514e0
    Author: Kittensx <[email protected]>
    Date:   Sun Oct 27 10:15:35 2024 -0400

        Merge pull request #1 from Kittensx/Kittensx-Simple-Karras-Exponential-Scheduler-2

        Added New Scheduler

    commit 78514e0
    Author: Kittensx <[email protected]>
    Date:   Sun Oct 27 10:08:52 2024 -0400

        Add files via upload

    commit e568ea5
    Author: w-e-w <[email protected]>
    Date:   Fri Oct 25 00:07:15 2024 +0900

        warning WebUI is installed under a dot directory

    commit 984b952
    Author: w-e-w <[email protected]>
    Date:   Thu Oct 24 22:05:51 2024 +0900

        Fix DAT models download (AUTOMATIC1111#16302)

    commit 5865da2
    Merge: bb1f391 c2ce1d3
    Author: w-e-w <[email protected]>
    Date:   Sun Oct 20 11:40:02 2024 +0900

        Merge pull request AUTOMATIC1111#16569 from AUTOMATIC1111/feat/ztsnr-auto

        Automatically enable ztSNR based on existence of key in `state_dict`

    commit bb1f391
    Author: w-e-w <[email protected]>
    Date:   Sun Oct 20 09:58:53 2024 +0900

        clarify readme: weget ... chmod +x webui.sh (AUTOMATIC1111#16251)

    commit 6a59766
    Author: w-e-w <[email protected]>
    Date:   Sun Oct 20 09:56:12 2024 +0900

        Add Skip Early CFG to XYZ (AUTOMATIC1111#16282)

        Co-authored-by: Yevhenii Hurin <[email protected]>

    commit 65423d2
    Author: w-e-w <[email protected]>
    Date:   Sun Oct 20 09:52:47 2024 +0900

        MIME type text/css (AUTOMATIC1111#16406)

    commit c2bc187
    Author: w-e-w <[email protected]>
    Date:   Sun Oct 20 09:51:59 2024 +0900

        fix modalImageViewer preview/result flicker (AUTOMATIC1111#16426)

    commit d0b27dc
    Merge: bb4cbaf cbaaf0a
    Author: w-e-w <[email protected]>
    Date:   Sun Oct 20 09:42:21 2024 +0900

        Merge pull request AUTOMATIC1111#16300 from hello2564/fix_NGMS_pr_typo

        fix NGMS pr typo

    commit c2ce1d3
    Author: catboxanon <[email protected]>
    Date:   Sat Oct 19 19:58:13 2024 -0400

        Automatically enable ztSNR based on existence of key in state_dict

    commit bb4cbaf
    Merge: c462e5a 9677b09
    Author: w-e-w <[email protected]>
    Date:   Sun Oct 20 08:27:12 2024 +0900

        Merge pull request AUTOMATIC1111#16341 from gutris1/devv

        add break-word for geninfo in pnginfo

    commit c462e5a
    Merge: 8b19b75 c9a06d1
    Author: catboxanon <[email protected]>
    Date:   Sat Oct 19 10:51:16 2024 -0400

        Merge pull request AUTOMATIC1111#16460 from AUTOMATIC1111/sd-1.5-url

    commit 8b19b75
    Merge: 907bfb5 1ae073c
    Author: AUTOMATIC1111 <[email protected]>
    Date:   Sat Oct 19 17:40:56 2024 +0300

        Merge pull request AUTOMATIC1111#16567 from AUTOMATIC1111/feat/sdxl-vpred

        Support and automatically detect SDXL V-prediction models

    commit 907bfb5
    Author: AUTOMATIC1111 <[email protected]>
    Date:   Sat Oct 19 17:33:58 2024 +0300

        add w-e-w and catboxanon to codeowners file

    commit 1ae073c
    Author: catboxanon <[email protected]>
    Date:   Sat Oct 19 06:53:19 2024 -0400

        Support SDXL v-pred models

    commit c9a06d1
    Author: missionfloyd <[email protected]>
    Date:   Tue Oct 8 16:50:39 2024 -0600

        Use stable-diffusion-v1-5 repo instead

    commit f57ec2b
    Author: missionfloyd <[email protected]>
    Date:   Tue Sep 3 19:58:29 2024 -0600

        Update stable diffusion 1.5 URL

    commit 9677b09
    Author: gutris1 <[email protected]>
    Date:   Wed Aug 7 17:37:23 2024 +0700

        add break-word for geninfo in pnginfo

    commit cbaaf0a
    Author: hello2564 <[email protected]>
    Date:   Wed Jul 31 14:55:30 2024 +0800

        fix NGMS pr typo

    commit 4823909
    Merge: 850e149 82a973c
    Author: AUTOMATIC1111 <[email protected]>
    Date:   Sat Jul 27 15:50:26 2024 +0300

        Merge branch 'master' into dev

    commit 850e149
    Merge: 834297b 8e0881d
    Author: AUTOMATIC1111 <[email protected]>
    Date:   Sat Jul 27 15:47:49 2024 +0300

        Merge pull request AUTOMATIC1111#16275 from AUTOMATIC1111/fix-image-upscale-on-cpu

        fix image upscale on cpu

    commit 8e0881d
    Author: w-e-w <[email protected]>
    Date:   Sat Jul 27 21:10:20 2024 +0900

        fix image upscale on cpu

        for some reason upscale using cpu will fail with
        RuntimeError: Inplace update to inference tensor outside InferenceMode
        switch from no_grad to inference_mode seems to have fixed it

    commit 834297b
    Merge: 9f5a98d c19d044
    Author: AUTOMATIC1111 <[email protected]>
    Date:   Sat Jul 27 07:09:08 2024 +0300

        Merge branch 'master' into dev

commit e3e3d1bbdd2f8ddd33c361fce27ddc4f26e3a4a7
Author: Thomas Ward <[email protected]>
Date:   Sun Oct 13 17:36:45 2024 -0400

    Update requirements.txt

    Add wheel to requirements because 'packaging' module is referred to and is not available unless you install 'wheel' in the environment.

commit e76d0a2
Merge: 4ea5403 2571d78
Author: Kittensx <[email protected]>
Date:   Sun Oct 27 12:07:45 2024 -0400

    Merge pull request AUTOMATIC1111#4 from Kittensx/Kittensx-patch-2

    Add files via upload

commit 2571d78
Author: Kittensx <[email protected]>
Date:   Sun Oct 27 12:06:08 2024 -0400

    Add files via upload

    forgot to include this in my lasst commit

commit 4ea5403
Merge: 6ffb728 da2e709
Author: Kittensx <[email protected]>
Date:   Sun Oct 27 11:08:53 2024 -0400

    Merge pull request AUTOMATIC1111#3 from Kittensx/Kittensx-patch-Simple-KES

    Update simple_karras_exponential_scheduler.py

commit da2e709
Author: Kittensx <[email protected]>
Date:   Sun Oct 27 11:08:06 2024 -0400

    Update simple_karras_exponential_scheduler.py

commit 6ffb728
Merge: 82a973c 78514e0
Author: Kittensx <[email protected]>
Date:   Sun Oct 27 10:15:35 2024 -0400

    Merge pull request #1 from Kittensx/Kittensx-Simple-Karras-Exponential-Scheduler-2

    Added New Scheduler

commit 78514e0
Author: Kittensx <[email protected]>
Date:   Sun Oct 27 10:08:52 2024 -0400

    Add files via upload

commit f3bb7e8
Merge: 82a973c c9a06d1
Author: kafffka-cat <[email protected]>
Date:   Thu Oct 10 01:18:03 2024 +0300

    Merge pull request #1 from kafffka-cat/sd-1.5-url

    Sd 1.5 url

commit c9a06d1
Author: missionfloyd <[email protected]>
Date:   Tue Oct 8 16:50:39 2024 -0600

    Use stable-diffusion-v1-5 repo instead

commit f57ec2b
Author: missionfloyd <[email protected]>
Date:   Tue Sep 3 19:58:29 2024 -0600

    Update stable diffusion 1.5 URL
@licyk licyk mentioned this pull request Nov 3, 2024
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@catboxanon catboxanon catboxanon approved these changes

@AUTOMATIC1111 AUTOMATIC1111 Awaiting requested review from AUTOMATIC1111 AUTOMATIC1111 is a code owner

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@w-e-w @n0kovo @catboxanon