Target image loop and sort

[ledahu]

Hi

Im trying to swap a female on an image (txt2img) with many Male in background. (sincity lora)
but, the reActor stop on the first male in background and say "not same gender" and stop the swap (gender female/female checked in the UI)

so i looked in your code (im noob in python, not in php, java, javascript ;) )

it seems that the script fail to continue on target_image loop if target_face >0 and first check fail.

but, debuging the object traget_image with a pprint, i saw that there is a bbox property, and it gave me an idea :

if bbox from insightface analyse is x,y,w,h , may it could simple and cool to get w * h, then sort target_image by face area surface, and then target firstly the biggest face => so no more problem in image with people in background, whatever the gender or age
=> can be an option in UI (biggest is foreground = index0)

no ?

[ledahu]

if it can help , here is the analyse_faces modified to set flags (in reactor_swapper.py)
isLeft, isRight, isTop, isBottom, isCentered, isBiggest and isMainChar

so it could be cool to have a check box : target is the centered character, the biggest , the one to the left etc.

for now, i don't know how to use them in your "operate", but im on it !

note: seems that bbox is 2 points and not x,y,w,h

from pprint import pprint
import math

def analyze_faces(img_data: np.ndarray, det_size=(640, 640), det_thresh=0.5, det_maxnum=0):
    logger.info("Applied Execution Provider: %s", PROVIDERS[0])
    face_analyser = copy.deepcopy(getAnalysisModel())
    face_analyser.prepare(ctx_id=0, det_thresh=det_thresh, det_size=det_size)
    # set the face_analyser on a tmp var
    tmp=face_analyser.get(img_data, max_num=det_maxnum)
    # reconstruct the image object to get width and height of the img
    im = Image.fromarray(img_data)
    centerX=im.width/2
    centerY=im.height/2
    minX=10000
    maxX=0
    minY=10000
    maxY=0
    centered=10000
    biggest=0
    #first loop to calculate and initialise flags
    for item in tmp:
        item['x']=(item['bbox'][0]+item['bbox'][2])/2
        item['y']=(item['bbox'][1]+item['bbox'][3])/2
        item['area']=(item['bbox'][2]-item['bbox'][0])*(item['bbox'][3]-item['bbox'][1]) #Area to sort
        item['distanceFromCenter']=math.sqrt((centerX-item['x'])*(centerX-item['x'])+(centerY-item['y'])*(centerY-item['y'])) #Distance from center to sort
        item['isCentered']=0
        item['isBiggest']=0
        item['isLeft']=0
        item['isRight']=0
        item['isTop']=0
        item['isBottom']=0
        item['isMainChar']=0
        minX=min(minX,item['x'])
        minY=min(minY,item['y'])
        maxX=max(maxX,item['x'])
        maxY=max(maxY,item['y'])
        centered=min(centered,item['distanceFromCenter'])
        biggest=max(biggest,item['area'])

    #second loop to set flags 
    for item in tmp:
        if(item['x']==minX): item['isLeft']=1
        if(item['x']==maxX): item['isRight']=1
        if(item['y']==minY): item['isTop']=1
        if(item['y']==maxY): item['isBottom']=1
        if(item['area']==biggest): item['isBiggest']=1
        if(item['distanceFromCenter']==centered): item['isCentered']=1
        if(item['isCentered'] and item['isBiggest']): item['isMainChar']=1

    #pprint(tmp)
    return tmp

[ledahu]

added a default behavior : if one source and multiple random target face in result => target is the centered one

in my previous code replace
return tmp
by
return sorted(tmp, key=lambda x: x.bbox[0])

so the analyse return the already sorted array by x position

then in operate (line ~800)

        if source_face is not None and wrong_gender == 0:
            
            # default face_num is keeped, but if there is random faces in target, find the index of the centered one and replace it
            custom_num=face_num
            if(len(target_faces)>1 and len(source_faces_index)==1):
                for i,item in enumerate(target_faces):
                    if(item['isCentered']==1) : custom_num=i

            target_face, wrong_gender, target_age, target_gender = get_face_single(target_img, target_faces, face_index=custom_num, gender_target=gender_target, det_thresh=detection_options.det_thresh, det_maxnum=detection_options.det_maxnum)

certainly to improve !

[ledahu]

this one seems to give good results

def analyze_faces(img_data: np.ndarray, det_size=(640, 640), det_thresh=0.5, det_maxnum=0):
    logger.info("Applied Execution Provider: %s", PROVIDERS[0])
    face_analyser = copy.deepcopy(getAnalysisModel())
    face_analyser.prepare(ctx_id=0, det_thresh=det_thresh, det_size=det_size)
    # get the face_analyser on a tmp var
    tmp=face_analyser.get(img_data, max_num=det_maxnum)
    # reconstruct the image object to get width and height of the img
    im = Image.fromarray(img_data)
    centerX=im.width/2
    centerY=im.height/2

    #Sort by X position
    tmp=sorted(tmp, key=lambda x: x.bbox[0])

    #loop to calculate properties
    for i,item in enumerate(tmp):
        item['x']=(item['bbox'][0]+item['bbox'][2])/2
        item['y']=(item['bbox'][1]+item['bbox'][3])/2
        item['area']=(item['bbox'][2]-item['bbox'][0])*(item['bbox'][3]-item['bbox'][1]) #Area to sort
        item['distanceFromCenter']=math.sqrt((centerX-item['x'])*(centerX-item['x'])+(centerY-item['y'])*(centerY-item['y'])) #Distance from center to sort
        item['weight']=item['area']/(item['distanceFromCenter']+1)
        item['pos']=i #Store the pos sort
        item['perfectCentered']=0
        if(centerX>item['bbox'][0] and centerX<item['bbox'][2] and centerY>item['bbox'][1] and centerY<item['bbox'][3]):
            item['perfectCentered']=1

    return tmp

def getIndexOfCenteredTarget(source_gender,target_faces):
    #Sort by weight 
    tmp=sorted(target_faces, key=lambda x: x.weight, reverse=True)
    res=0;
    for i,item in enumerate(tmp):
        if(source_gender==item['gender']) : 
            res=item['pos']
            break

    return res     

        if source_face is not None and wrong_gender == 0:
            
            # default face_num is keeped, but if there is random faces in target, find the one centered and replace it
            custom_num=face_num
            if(len(target_faces)>1 and len(source_faces_index)==1):
                custom_num=getIndexOfCenteredTarget(source_face['gender'],target_faces)
                
            target_face, wrong_gender, target_age, target_gender = get_face_single(target_img, target_faces, face_index=custom_num, gender_target=gender_target, det_thresh=detection_options.det_thresh, det_maxnum=detection_options.det_maxnum)

[ritlo]

Thanks, I was looking for this and it works pretty good. I think you should try posting in issues or a pull request, especially since reactor comfyUI node does Large>small faces for its face index but for some reason the webui plugin does left to right and it doesn't work good.

[ritlo]

I found this thread via google, I didn't even know github repos have discussion section, seems like your fix has gone unnoticed because of this.