-
Notifications
You must be signed in to change notification settings - Fork 337
How to Handle Builds Where More Than 10k Pages Change? #1191
Comments
I’m unfamiliar with Rust but it appears that
Is there a way that wrangler can handle deletes the same way it handles uploads? (It easily uploads more than 10k files) This is blocking me from using this in production. Thanks. Really loving Workers/Worker Sites and stoked to be able to use them and sing their praises to the SSG community. |
hey y'all! this limit does appear in both api endpoints; i think it would be easy as pointed out to "batch" deletes the way we batch uploads. PRs welcome, but I'll have a look at this Friday if no one has taken it on. |
@nickreese the hack in the meantime would be to rename the existing site namespace in the dashboard; this is as easy as appending/prepending "old" or something to the title of the namespace. for sites, wrangler uses a naming convention for the site namespace to decide whether to create a new namespace or push changes to the old one, so if you rename your old namespace it will stay bound to your worker in production (it only changes the title, not the id which is how we bind namespaces); then when you run wrangler publish again, wrangler will look for the namespace by its old name and when it doesn't find it there, it'll create a new one, upload your assets to that, and bind the new namespace to your worker. once it's done publishing you can delete the old namespace. |
@ashleymichal Thanks for looking into this. I dove in this morning and wrote my own upload process in node.js that matches Wranglers upload process. (Hence my comment over here.) All that said, I would love to be able to use wrangler instead of my own hacky code to get the job done. Thanks! |
This comment has been minimized.
This comment has been minimized.
Hey @ashleymichal , did you get a chance to look into batching deletes? I'm currently using the hack you describe (prepending namespace so Wrangler creates a new one) but it would be awesome not to have to worry about this when we go to production. Thanks again for looking into this! 👍 🙏 |
Any update on this issue @ashleymichal? This is also preventing us from using Workers Sites in production, and it would be great to know if it will be solved in the near future. |
there is a lot of work for cleaning up sites, but this needs a priority bump. we'll get this in for 1.10.0 |
I'll take a look at this issue |
Thanks for this workaround. Works like a charm. |
🐛 Bug Report
My SSG generates new pages each build. The first
wrangler publish
always works, but subsequent ones fail due to the 10k key limit.Number of keys to delete (17397) exceeds max of 10000
Is there a workaround for this besides deleting the store ahead of time? (This works for staging but not production).
- yes | wrangler kv:namespace delete --namespace-id={ID for __static-staging-workers_sites_assets}
This said, I was reading the code for for
getAssetFromKV
and it appears that a possible workaround would be to:1 - Create a new KV store for the content.
2 - Specify wrangler to use that store for the content.
3 - After deploy delete the old KV store.
Is there a way to do step 2? I've read all of the docs and can't find anything about how to specify the store for the worker to use.
Environment
Steps to reproduce
1 - Build a project with > 10k pages.
2 - Make a change to more than 10k of those pages.
What did you expect to see?
The build be successful.
What did you see instead?
Number of keys to delete (17397) exceeds max of 10000
The text was updated successfully, but these errors were encountered: