Option to add ControlNet image file name to prompt when batch processing?

[FugueSegue]

Is there a way to add the text of file names from the Input Directory of a batch to the respective prompts for each image? I think such an option would would be especially helpful with the reference-only preprocessor.

I am trying to generate hundreds of images that will subsequently be used as a dataset for a style. I want to experiment with mixing various styles in order to do this. The reference-only preprocessor makes this task easier.

For example, I have three pictures that I want to batch process in ControlNet. One is a photo of a car, another of a truck, and another of a bus. The files are named, 01_car.png, 02_truck.png, and 03_bus.png. For each image that I generate, I want to combine three art styles. I can specify "[art by van gogh|art by rembrandt|art by alphonse mucha]" in the prompt but the reference-only preprocessor requires something in the prompt that describes each respective image. Otherwise it makes some sort of random guess as if I had left the prompt field blank. When the ControlNet reference-only preprocessor uses the 01_car.png file in the batch, I need to explicitly state in the prompt that it is a "car".

What I need to have it do is generate three images. Each image should be generated with these three prompts and ControlNet conditions:

Image 1 (a car)
Prompt: car, [art by van gogh|art by rembrandt|art by alphonse mucha]
ControlNet: reference-only using an image of a car.

Image 2 (a truck)
Prompt: truck, [art by van gogh|art by rembrandt|art by alphonse mucha]
ControlNet: reference-only using an image of a truck.

Image 3 (a bus)
Prompt: bus, [art by van gogh|art by rembrandt|art by alphonse mucha]
ControlNet: reference-only using an image of a bus.

I attempted to use the Prompts From File script and the Dynamics Prompts extension to accomplish this task. I even tried implementing Jinja2 templates. But in each of those cases it applied all the permutations specified on each individual image of the batch before moving on to the next. That's not what I want. I want the ControlNet batch process to perform as I described above.

I suggest a solution to this. Add the words in each file name in the batch (excluding numbers and symbols) to the respective prompts of each batch image. Perhaps a checkbox near the top of the tab that is labeled "Add Filename to Prompt". Or a drop-down menu that has options to add the filename words to the beginning or end of the prompt.

One of the primary concerns for training a unique art style for stable diffusion is curating a vast dataset of images that have examples of various things that are rendered in that desired style. For a few weeks I have been attempting to randomly generate images by specifying a subject in a prompt along with an experimental mixture of styles and then choose the best ones to use in a dataset. This is difficult because of the limited range of various art styles. Prompting "space shuttle, art by van gogh" may or may not have desirable results because there are no artworks by Vincent van Gogh that depict spacecraft. With ControlNet, and especially with the help of the reference-only preprocessor, it's now much easier to imbue any image with a specific style.

I believe the reference-only preprocessor is the key to generating wide-ranging datasets for style training. But the batch process for reference-only needs some way of specifying the subject of each image.

Or is there another way to do what I'm trying to do? I searched for similar discussions and didn't find anything related to my idea. Please forgive me if I am in error.

[FugueSegue]

Is this possible? It's been three weeks and I've gotten no response. I think this is a really good idea.

[Chubbly]

I think this would be a great addition too. I am convinced I am missing something because it feels so obvious that this information would be available somewhere that I haven't raised a feature request issue yet but perhaps it really just doesn't exist at the moment?

[huchenlei]

I think the issue described here can be achieved using the API.

You can have a custom script sending requests to WebUI with your desired ControlNet input and prompt.
See https://github.com/Mikubill/sd-webui-controlnet/wiki/API#integrating-sdapiv12img for more details on how use the API.

[Chubbly]

While it is true that using the API can offer a solution to this issue, it would be way nicer if ControlNet just had an option to include the filename as part of the batch process gen parameters.

I know it sounds entitled to ask but it feels like this should be easy to add as ControlNet definitely has the file name information when it is running a batch process, it just needs to pass that info along when it is outputting gen info that it already appends to every image. Is there a complication here that makes this more difficult than it appears?

[huchenlei]

I am not familiar with the batch process code. You probably want to ask @ljleb.

[FugueSegue]

When I proposed this a month ago I also posted this idea elsewhere on Github and on Reddit. I think it was someone on Reddit who brought up the idea of using the API. Just as you did. I was also given the same link that you provided. I was reluctant to try it because I had never written any code that interacted with an API before.

It was easier than I thought. I have experience writing script for Python but I can't call myself an expert with that programming language. I examined an API script example and tried it. Once I got it to work, it gave me ideas of what to do. For parts of the example code I didn't completely understand, I discovered ChatGPT was extremely helpful.

I am now able to write Python script that accomplishes what I wanted to do. I am so pleased with the result that I am beginning to design my own app that I might use in my workflow in Photoshop. If I develop the app into something useful, perhaps I will share it with the community.