Parametric Prompting

[swyxio]

a lot of people are exploring latent space by parameterizing their prompts:

• https://twitter.com/_benoitmartinez/status/1566439265218838530?s=21&t=RdQ-vfnKdUWZ-oZJhbc37g
• https://twitter.com/xsteenbrugge/status/1566957660200632323?s=21&t=q-puVhdTmExlbvDgNdMAqw

i would like a way to set these up in a comma separated list and then basically leave SD running with sd-buddy managing them in sequence. what do you think?

[breadthe]

I like this a lot. I was thinking something along those lines, but more like batching a set of prompts. It needs some brainstorming.

[swyxio]

i think we might need to agree on what a ui would look like.

if you look at this part of how jasper.ai works https://youtu.be/loRJkgeRgt4?t=228 (timestamped) you can see people are making templates and then filling in with variables

maybe we could do that? with separate UI for the variables

[breadthe]

I like the first tweet you linked. So you're saying to come up with a way to parse this kind of description and replace the vars with values from an array of sorts?

With a list like this for jobs

let job = ["doctor", "firefighter", "sailor"]

You would replace it here randomly

"_age_ _adjective_ _job_ _bodyType_ _genre_ from _country_, _adjective_ lighting"

To get something like

"25 <adjective> doctor <bla> <bla>"

Can you mock up a UI in https://www.tldraw.com/ or similar?

The logic could be done as standalone code in JS, outside of the app, to integrate it later.

As for the integration, it needs a queuing system with the ability to set max cycles (so it doesn't run forever).

[breadthe]

Here's a REPL for a POC.

https://svelte.dev/repl/f5eacf80f01b4a99b1c5199f0957cf71?version=3.50.0

It mostly works but only for 1 parameter. Maybe you can figure out what the problem is, I'm a little short on time.

[swyxio]

your problem was this line

// replace all the $vars in the prompt
const promptString = prompt.replaceAll(`$${customVar.name}`, val)		

means you start over from the prompt template each time. let me try to fix:

• repl: https://svelte.dev/repl/fc10be08e0724a73aa7bad15f46d4a28?version=3.50.0
• screenshot: https://user-images.githubusercontent.com/6764957/189250765-72078b07-5bc0-4741-8b90-8a25d416d14d.png

	function buildStrings(customVarsArray) {
		const vars = customVarsArray
		let newPromptStrings = null;
		
		if (!vars.length) return
		console.log(vars)

		// loop through all the vars
		vars.forEach((customVar) => {
			// if this is the first loop, init the promptstrings array with the blankest version of the prompt
			if (!newPromptStrings) newPromptStrings = [prompt];
			const temp = []
			for (let partialPrompt of newPromptStrings) {
				for (let val of customVar.values) {						
					// replace all the $vars in the prompt
					const promptString = partialPrompt.replaceAll(`$${customVar.name}`, val)		
					temp.push(promptString)
					console.log(promptString)
				}
			}
			newPromptStrings = temp
		})			
		// you can randomize later if you want
						
		// save it to the store
		promptStrings.set(newPromptStrings)				
	}

[swyxio]

queuing - I'm talking about the front-end stuff

ah ok. i think if you are able to wrap the cli run in a new Promise and only resolve when the run is done, you might be able to manage the whole thing as async await in-memory.... this would be a temporary implementation, a more "serious" system would persist this queue to disk so it can restart if interrupted halfway, https://github.com/OptimalBits/bull would be the "upgraded" version

randomize - if you set the seed to -1 it will use a random one

ohhh i didnt realize that, in that case forget my suggestion, just make (-1 for random) more obvious and then store the seed used

In other news, have you figured out how to run the command in a deterministic way? I don't know what parameter combo works to produce the same image each time. I know it's possible, I just haven't done any research on it yet.

i think just make sure the seed, cfg, steps and prompt are the same? (i'm not sure if iter is needed) . also useful to note the VERSION of Stablediffusion and the associated checkpoint model being used, as that is also a moving target

[breadthe]

Great, let me play with your solution and the queuing concept.

I was thinking that queuing can be done in 2 ways. The easiest is to put all the strings in an array in the store. Obviously this won't persist if you refresh/close the app. So the next step is simply to store it in localStorage the same way I currently store the runs.

In the future I want to move the runs to something more accommodating like Indexed DB, but I think the queue can stay in localStorage.

Re: randomize. While -1 uses a random seed, it doesn't save the value anywhere and it doesn't tell me what it picked either. So I'll have to generate the random number myself, between the allowed ranges.

[breadthe]

Re: // you can randomize later if you want I just realized I don't need to randomize the parameters after all, as long as I have all the variations.

[breadthe]

Also take into account the new UI layout that I added in v0.6.0.

[swyxio]

for those who find this, v0.8 has this feature now https://github.com/breadthe/sd-buddy/releases/tag/v0.8.0

[breadthe]

Put this feature through its paces if you get a chance. It's possible there may be some buggy behavior. I noticed some anomalies in dev mode but it was inconsistent. I decided to release it anyway rather than to spend forever hunting ghosts.

[breadthe]

Parametric prompt randomization

1. What is the purpose and/or main benefit of randomizing prompt order? To me personally it doesn't really matter which order they get generated, since the end result is the same. Maybe I'm missing something though.
2. Simple randomize or special algorithm? By simple I mean pick a random index between [0..array.length] and pop that prompt out.
3. Would it make sense to add a checkbox for "Randomize generation order" or similar? Or just randomize implicitly?

[swyxio]

1. because it's very easy to create 100,000's of prompt combinations, sometimes i dont really care about running them to completion, i just want to sample randomly from that prompt space
2. simple random works i think
3. imo randomize implicitly is best but your call