New features and feature requests

[C0untFloyd]

Here I would like to collect and show planned features for roop unleashed.
My wishlist so far would be:

• More configuration for blending faces (bbox manipulation, blur, erode etc). Get rid of pesky hairlines!
• Other text masking options, perhaps adding slow but perfect masking for still images (Grounding Dino, SAM)
• Cleanup and improve the processing workflow
• Rewrite enhancer code. Especially codeformer re-invents the wheel currently with it's own face detection etc.
• Improve the extras tab to have basic but handy video cutting functions, perhaps batch text-masking if so desired
• Investigate on hair or complete zero-shot head traversal. One can dream...
• Have a plugin system to support other deepfake models e.g. Ghost
• Processing only parts of a video

[lysxelapsed]

• Investigate on hair or complete zero-shot head traversal. One can dream...

not sure what you mean by that exactly, but sounds interesting ;-) can you elaborate further? excited about further config-options for blending etc., too!

well, you know what's on @anxiousottergames ' and my wishlist.. ;-p but like I already said: you focus on what you deem necessary, first. it's coming along nicely :-)

[C0untFloyd]

not sure what you mean by that exactly, but sounds interesting ;-) can you elaborate further?

With Deepface Lab you can swap a whole head but this needs a lot of training. Having the possibility to transfer at least hairstyles would be great I think...I toyed around with this but it's unusable slow.

[lysxelapsed]

bummer.. maybe something faster will show up eventually.

[Fijitrix]

Sounds good to me. I'm not sure if you had them listed in order of importance but I personally feel like even the order of the planned features listed is perfect.

[JeetGuhaThakurta]

With Roop development now shut down, this is the only place where we can expect improvements on this nice tool. After a lot of tests, I figured out (at least within my test results) that the biggest flip side is the eye movement. It is totally missing in a video swap. The model seems to be looking straight forward, irrespective of the actions. Hence, the model looks like visually impaired or blind.

This is not noticeable in a still image swap because it is a still image, there is no action. But in a video, this lack of eye movement is very easily noticeable. You can tell within 4-5 seconds that the video is fake. Above all of the other enhancements discussed, I think we first need to focus on how to copy the eye-movements of the target to the output while we swap the face from the source.

Basically, the face will come from the source, we need to detect the eyeballs, copy the eyeball direction from the target (not using the source eyeball directions). That will add life to our end product. This technology is already there because Adobe Photoshop can do this. We can change eye direction in a photo using Photoshop. We have to do that same thing with the output image after detecting the direction from the target frames. But I am not sure if there is an open source project already available for this. A close match would be Gaze Correction, but it has limited features. It may be used as a start point.

[C0untFloyd]

That's why I linked https://github.com/ai-forever/ghost which especially cared about gaze and emotions. From my limited testing it did this better than the rest but the face mask was lacking or rather looked too artificial.

[JeetGuhaThakurta]

What I am doing right now is, I am using multiple source images having various angles. This is not difficult to do, if we have a source video and we extract several source frames before the swapper is called from main. Then in the swapper, I am selecting the best match (from the source image set) based upon the face angle of the target frame. This is giving a better result, at least there are eye movements. But these movements are not fully realistic, because they are not entirely driven by the target. Some half-way solution until we discover a better approach. I will look into Ghost and see if we can borrow just the eye tracking part from there and add a new processor between the swapper and the enhancer to rectify the gaze. Thanks for the leads!

[aripsam]

Can you share the code for this? Would love to try

[C0untFloyd]

based upon the face angle of the target frame

This is actually quite a good idea, perhaps even better than the blending of several images.

[patientx]

can we get a fps changer (probably doable with ffmpeg even without reencoding) this is useful for some high resolution videos that are 60fps and because of this takes a lot of time but can be 30fps cutting processing time in half , maybe processing every second frame instead of every frame ?

Also like in stable diffusion extension can we select faces by number they appear on screen ? (1 and 3 for example out of 4 faces)

Both of these are especially useful for 180vr videos where a persons face appear two times but since they are in different angles app doesnt recognize both as the same almost always. This is not a problem if only one person is in the video just select all faces BUT if there are more than one person on screen we either need to change all faces or just one persons face on one side of the video.

[C0untFloyd]

I tested Fast Segment-Anything yesterday and am quite disappointed. It's faster than the current Clip2Seg but surprisingly very bad when using low-resolution inputs. If it can identify objects, the resulting masks are way better though. I don't know if it's worth the hassle.

[lost-in-emotions]

Suggestion on the Side: What about an Upload Source for an Audio File (mp3) for "directly" a new Lip Sync? -> Overwrits the Lip Sync of the Target Video and Change it with the "Manuellextra located Audio File.

:-)

[C0untFloyd]

Hm...uh...I don't know. With an audiofile of the exact same size this is very easy of course but I don't want to open a can of worms, because of different audio formats, weird sizes, special ffmpeg handling. I'd rather leaves this to other great, specialized tools like this

[lost-in-emotions]

but isnt it lossless-cut only a editing programm?

you can limit the usage data of the audio files in roop. f.e. so only mp3 is allowed, or something.

sadtalker is really nice, but only can lipsync to a single file/frame. i made my cosuin a birthday present with his face on a famous movie scene, where the actor sing a song. i used another song and cut it together in after effects. but the lip sync was - of course wrong (on the old/original audiofile. so here it would be perfekt if i could adjust the lipsync with an audio file of my desire

[C0untFloyd]

but isnt it lossless-cut only a editing programm?

Yes and adding a custom audio track is editing. From its description:
Combine arbitrary tracks from multiple files (ex. add music or subtitle track to a video file)
Remove unneeded tracks
👉 Replace or re-encode only some tracks 👈
Extract all tracks from a file (extract video, audio, subtitle, attachments and other tracks from one file into separate files)

The idea with your cousin is really neat but I won't do it.

[sordidloam]

Controlnet?Sent from my iPhoneOn Aug 21, 2023, at 6:24 AM, JeetGuhaThakurta ***@***.***> wrote: With Roop development now shut down, this is the only place where we can expect improvements on this nice tool. After a lot of tests, I figured out (at least within my test results) that the biggest flip side is the eye movement. It is totally missing in a video swap. The model seems to be looking straight forward, irrespective of the actions. Hence, the model looks like visually impaired or blind. This is not noticeable in a still image swap because it is a still image, there is no action. But in a video, this lack of eye movement is very easily noticeable. You can tell within 4-5 seconds that the video is fake. Above all of the other enhancements discussed, I think we first need to focus on how to copy the eye-movements of the target to the output while we swap the face from the source. Basically, the face will come from the source, we need to detect the eyeballs, copy the eyeball direction from the target (not using the source eyeball directions). That will add life to our end product. This technology is already there because Adobe Photoshop can do this. We can change eye direction in a photo using Photoshop. We have to do that same thing with the output image after detecting the direction from the target frames. But I am not sure if there is an open source project already available for this. A close match would be Gaze Correction, but it has limited features. It may be used as a start point. —Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[C0untFloyd]

I don't know how or why ControlNet should be added to a face swapper? This is no diffuser and no inpainting so far...

[marlonio]

• More configuration for blending faces (bbox manipulation, blur, erode etc). Get rid of pesky hairlines!

I'm searching for the possibility to erode and blur the face mask but on the last roop-unleashed (3.3.4) I can't find the options, is that possible? I'm asking that 'cause i'm still obtain a slight ghost-box around my swapped face and i want to improve the results blurring the box a bit.

[Tolga077]

Since I had a potato computer, I integrated gpen 256 thinking it was better than nothing. But I couldn't get the Original/Enhanced image blend ratio part to work correctly. It only works between 0.90 and 1. When it goes below that, it adds strange blurs and distorts the image. what would be the reason ?

[lysxelapsed]

did you just replace the 512 model with the 256 version, or have you made changes to the code? my guess would be, that (at the very least) you have to adjust Enhance_GPEN.py at line 23 (model_path = resolve_relative_path('../models/GPEN-BFR-512.onnx')) and 34 (temp_frame = cv2.resize(temp_frame, (512, 512), cv2.INTER_CUBIC)). can't say if that's the only thing you'd have to adjust (probably not 😆) - you'd have to ask @C0untFloyd (still on vacation)

[eobard1991]

@C0untFloyd quick question. I noticed the TEMP folder gets really chunky after extended use and keeps facesets duplicated from the UI. Can i periodically delete the contents of the temp folder?

[lysxelapsed]

yes you can, there's even a button for it in the settings tab.

[C0untFloyd]

Transfered here from the discussion started by @aripsam

Since this version is pretty stable and has been running well for a while now, I'm just wondering if there are any plans to add any more features? Anything in the pipeline?

Some features I can think of -

• offset face sides, like how you have top and bottom
• mask specific face areas like nose or lips (is kinda handled with the clip but wondering if there's anything new)
• adjustable blending settings
• saw something in the dev branch around face rotation to improve quality
• for colab, would be nice to point to a path on gdrive to access the target file and save output file

[C0untFloyd]

Well, it's not dead yet but admittedly development slowed down a lot because of the holidays, a lot of real life work and I still did not fully recover from my leg injury. I merged the PR for auto rotation into the dev-branch and would like to release this into main but before that I would like to improve this to rotate every face instead of the whole image. This isn't quickly done, as the PR is huge and I want to get it right. I also integrated a VR mode for swapping faces in VR images/movies but this too needs some more work to cope with fisheye distortion etc.
Then there is the new iteration feature, where the same face is blended multiple times together to restore even more of the face identity. Up to now I did not get it to work and it gives me headaches.
2 more additional features I would like to have:

• example images to see how new images in a faceset influence the swapping result. Preferably the example images should be loaded from a folder so you can supply your own.
• a way to save and load a complete "swap set" containing all source and destination faces

About your requested features, the first 3 were probably inspired by Rope? 😄
When I implemented the face offsets I could have easily integrated the sides as well, This however would clutter the GUI even more and from my experiments it's mostly unnecessary + doesn't work very well. I checked out the latest Rope version some days ago and I really wasn't able to do a simple face swap with it because I couldn't find my way through the billions of settings there! I'm trying to prevent feature creep myself, although I'm not always successful.
Face parsing was requested a lot over time but what is the use case? I mean if e.g. half of the face is occluded then the face detection would fail in the first place and without face there's no face parsing. The good thing about clip is you're not limited to face parts.
Blending can already be adjusted but perhaps you're talking about blur amount and erosion? Similar to the previous, easy to include but hard to get right with the limited web framework.
Colab: several people were reporting that it's not working anymore, didn't try myself though. Useful suggestion, which could be added to the settings tab.

[lost-in-emotions]

i also have a suggestion, maybe its caused by its limitation?
when i render portraits in SDXL with a face aprx. 50% of the image,
and then i use a face swap the result is very bad, very pixelated.
maybe its limited by the resolution?

[C0untFloyd]

Insightface Swapper Resolution. Please see the 2nd issue to ever grace this repo 😄 >>--> #2

[C0untFloyd]

Btw. I forked a new promising hairstyle transfer repo today, which would be a great addition to roop unleashed once it is working:
https://github.com/C0untFloyd/HairCLIPv2_UI

I can't get it to work so far though, not in Colab and not on my machine. Perhaps one of you can enlighten me, the problem is the dynamic compilation of python modules which I can't seem to get right. Also there is a lot of hopefully unnecessary Jupyter stuff in it, this would have to be removed for our own local needs.

[Elise96nl]

I wish my knowledge could be of use. First time ever i wish i could do some real programming, since i started stable and now your roop unleashed, i'm slowly learning how programming works, python is great.

Fun fact, the OG developer of python lives in my street (Guido van Rossum)

[C0untFloyd]

Oh just saw this today. Can't say that Python is my favourite but it's very easy to pick up and learn and the majority of machine learning apps is written in it. Actually I only started the whole roop and bark adventure to learn python, my daily bread and butter is C# 😄
Anyway, I've gotten the hair thingy to run since then but it's way too slow to be included in roop. You can read more about it here -> #624 (reply in thread)

[Elise96nl]

This app is so great. My new addiction. Only problem in my path to achieving perfection are the mouths. Not going to mask a mouth manual frame by frame, im doing 40k frame scenes. Clip2seg with trigger word: Open mouth does the trick, but the whole process falls back to cpu on my AMD system. So we go from 4 frames per sec, to 10 sec per frame. :P

I found this screenshot at

" It also comes with a 'Mouth Mask' mode that allows the mouth to interact with objects."
https://github.com/iVideoGameBoss/iRoopDeepFaceCam/blob/main/MouthMaskDemo.gif

This is the only thing roop unleashed is missing, a specific mouth masking tool.

[C0untFloyd]

Thanks for the link, this is so simple stupid but effective, I'm still giggling while doing experiments with it. You know, the way it works is just cutting out a box fairly the size of the mouth area but expanded downwards a lot, to include the chin region and more "obstructions". The mouth region is already known from the face detection, this isn't using another fancy AI model or segmenting it's sort of a poor man's mouth segmentation and very fast (although always running on cpu)! I already ripped the code and adapted it for my roop version, I'm pretty sure it will be in the next release 😃
You could achieve almost the same btw. by using advanced masking and offsetting the bottom to be higher than the mouth.

[Elise96nl]

@C0untFloyd you my friend, are awesome. Looking forward to the next version. If i can help in anyway (write wiki, test dev, etc) let me know!

[C0untFloyd]

Sure, I wouldn't mind if someone could add the missing pages to the wiki. It is free for all, everybody can contribute!
I experimented some more with the mouth restoration. For almost perfect results one still needs to use 512 pixel upscale and DFL masking. And it looks weird when target and source biometrics are very different, this is also very apparent when looking at the original repos example:

It doesn't look quite right to me but of course it's a step forward and I also turned it on for the live-cam tab.

[Elise96nl]

So now i have a few really awesome video's made with roop-unleashed. Sometimes a short video takes like 2 hours of tweaking to get perfect. Sometimes restoreformer is magic, sometimes gpen. And the values are different for every source.

I didnt keep a template/settings journal so now i want to re do some things i've made with different faces i'm stuck tweaking and trying again. Could we have like an settings/options to filename thing? or maybe like stable diffusion that we add meta data for each run.

something like this would be awesome:

Filename Output Template
{file}{time}{maskengine}{upscalevalue}{post-processing}{blendratio}__

[C0untFloyd]

These output templates already exist so far:

{timestamp} -> date and time appended
{i} -> index in list (number)
{file} -> original filename without extension
{date} -> date only
{time} -> time only

If you're too lazy to sort your experiments, well I'm too lazy to implement more placeholders 😜

[Elise96nl]

haha you are absolutly right. The date/time is nice, i will make an excel sheet with all the settings i use per run.

[ali-m-github]

Hello! Is it possible to allow drag and drop on this menu when a photo is already used in the target file field instead of having to click on close photo to add a new one?

Like the photo above, if I drag and drop a new photo it won't replace the photo in use but it would open it in a new tab, is it possible to replace? Would save a lot of time when doing many photos.
Thank you!

[C0untFloyd]

You're not limited to just 1 file, in fact you can batch process complete folders. Multiselect a couple of files and drop them all into that component.

[ali-m-github]

You're not limited to just 1 file, in fact you can batch process complete folders. Multiselect a couple of files and drop them all into that component.

Not that, what I mean is a lot of times I drag photos from Google or some sites directly to the interface so it would be good to be able to drag a new photo to replace the current target photo directly without having to click "X".

[C0untFloyd]

Unfortunately I can't actively access the UI components so this impossible.

[Elise96nl]

Hi @C0untFloyd

i have the following request or question.

im now doing batches of pictures. With multilpe persons on them, but every picture has different people. On the left side i've added 10/15 different face sets. And what i would like to do is do big batches with multiple faces without having to put the faces in the target by hand for every picture.

So an option to:

input = face A - Face B - Face C

and for every picture find all faces

if fround 2 faces on picture

automaticly do

face A to found face A - Face B to found face B

If this is to hard to implement. A way to change the first found from left to right to right to left or to 2nd found left to right would also fix my issue. I can then do the batch of pictures more then once.

so run 1 would swap face (first found left to right)
run 2 would swap face (2nd found left to right)
rund 3 would swap face (3th found left to right )

atm the only option is manualy select faces for 1000 pic's, or do all faces and have 2 or 3 clones :P

[Elise96nl]

i've been trying for 6 hours :P had it working with first and second. but now it's broken :( Im not a programmer haha

[C0untFloyd]

This is something which is repeatedly wished for since the beginning, I dug up this old issue #83 (I'm pretty sure you can find even older threads).
Anyway, this is a sort of tedious QoL feature where I'm just unsure how to do this with the current gradio (dis)abilities and I myself don't have any use for it. Can't I just do a mode like "All input faces in order". That would swap all input faces onto every unique face from left to right until it's running out of input faces. Deal?

[Elise96nl]

all input faces in order would be good!

If you cant do it i'm even more prooud i had it working with first and second found. But i did just swap the left to right to right to left.

by chaning this:

return min(faces, key=lambda x: x.bbox[0]) to return min(faces, key=lambda x: x.bbox[1])

at line 49 @ face_util

but where it broke down was

rotate image and re-detect face to correct wonky landmarks

            rotface = get_first_face(rotcutframe)

i had to insert 2 options here rotface = get_second_face etc but i didnt get that to work.

Just an option with more input faces. and more faces on a picture it not uses the same faces but just swaps ABC until there are no faces left would be great:

as you call

All input faces in order!

[C0untFloyd]

Done! Check it out and also try out the new virtual cam mode which is quite fun 😃

[Elise96nl]

Feels pretty hardcore. But i have to take a plane for work. (DJ around Europe) So im now "roop'ing" on my laptop by remote desktop connection on my workstation at home at the airport lobby :P Thanks for the quick upgrade. Will try it tommorow when im home and will let you know!

[neil10000000]

Hi @C0untFloyd

I have a feature request. It is kind of a quality of life update.

So when I choose Specify face selection for swapping to "selected faces" and set the action on no face detected to "skip frame" there is a lot of flickering in faces at weird angles.

could it be possible to remove the entire frame length if there are lot of fickering frames in between good continuos frames.

example - Suppose out of 10 frames 3 frames are skipped because no similar face was detected. I would love to skip those entire 10 frames all together so i dont see any flicker. Creating a smooth video without. I would rather have a portion of video frames to be cut out with a solid transition then have flickering faces frames that ruins immersion of the whole video.

I wolud love to keep continuos good frames that dont have any skipped frames between them. This will eliminate flickering issue on many of my projects.

A slider would be great that lets you set the number of skiped frames before the entire length of 10,20,30,40 etc continuos frames which ever the number you set in the UI are discarded.

If i set slider to 2 then it will discard the entire length of 10 frames if it encounters 2 skipped frames in between those 10 frames. eliminating the flicker.

It can be hard to explain what i mean but i know you will understand what i am trying to say. like in a video if i have 5 seconds of good face swap and then 2 seconds of face swape that contains many skipped frame then i would love to skip all those frames in those 2 seconds and only keep those 5 sec of good swaped frames.

Thanks

[C0untFloyd]

Yes I do understand and your reasons are perfectly valid. However this isn't as easy implemented as the other modes. You know, I created "Use last swapped" for almost the same reasons. That one counts the number of new unswapped frames and puts the last one into the current one, if no face is detected. If the count reaches 15 unswapped frames it will continue using the actual unswapped until a swapping occurs again. This has the advantage that the number of frames doesn't change (so you can still use audio) and that there is much less flickering. A big disadvantage though (and it will be the same with your option), motions are stopped in-between or are left out and this is almost more noticeable than flickering 😢
Now for the technical reason: all the current modes can be done during the processing of the video. Yours need to be done as a post-processing step, because you have to look over a timespan and cut out sequences which are already swapped and written into the file.
I dunno if it's worth the hassle. There was a discussion somewhere here about using DaVinci to remove flickering and other artifacts. Perhaps try that first?

[neil10000000]

So you created last swapped mode for this reason, hmm I thought as much. After trying it for a few times I never really liked the output it created and have never used it since. It skips frames quite often even within a span of a second or two which feels unnatural just like how flickering is. It creates a new kind of immersion breaking problem.

Its just my opinion that instead of using last swaped mode, I would rather use a post processing option in which i could specify how many seconds of continuos face swap frames clip I want and discard all other segments. I would rather have a 5 seconds clip with no jittery and flickering faces that feels more natural and brakes less of my immersion.

sure it comes with a downside that it breaks continuos movement and creates problem with the audio mismatch but it feels real and you dont get any feeling of it being fake. For audio we can just add a separate music track to the file to somewhat mitigate the issue

what my main point is sure if I feed roopunleased a 40 minute video and get 5-7 minute of continous face swaped frames, those 5-7 minute of continous swaped frames make the video feel real.

All comes down to personel preference - would you have continous movment video with flickering faces and skipped frames or would you have a continous face swapped frames without any flickering faces and skipped frames in between.

Sure it might involve an extra post processing step but once that video is ready you can use the output of that video and put it back into the target file and use it with other faces that is what i like to do.

A option for the same would be great though for the people who want to use it but its upto you ;)

By the way hope you could also implement batch processing of videos with different input faces been hoping for that feature for a long time. Sure we can do one face swap to multiple videos but it would be great to do different faces batch process on a single video.

Example : I would love to set 20 faces on a single video, go out and come back in evening and find 20 videos done. I would love it. Sure if you do ever implement this feature do give a option to set a timer to give break to the gpu in between videos or my 3070ti will get on fire by the time i come back to house :D

I would look into davinci as per your suggestion and see it it helps. that is all i wanted to speak about.

Its a great piece of software though. Have fun...