Video Tutorial:  https://www.patreon.com/posts/v3-0-lipsync-fix-99387166

(See the Thumbnail Video)

.

.

Renders

1) https://www.youtube.com/shorts/eqgTnq6azVI

2) https://www.youtube.com/shorts/ub9AlS1X6dU

Sources:

1) https://www.instagram.com/reel/C3FyWgYIc_x/

2) https://www.instagram.com/reel/C3p4ivKOl9S/

.

.

New Workflow!!

Heyyyy Everyone, A new Folder Has been added in the Drive Link, Containing 3 New Workflows:

https://drive.google.com/drive/folders/1z5N8rHwTRd8_P0vJBqlQy8mECNKX_6am

4_0 - LipScync Swapper - ReActor

A New Method of Swapping out - Lips, Eyes, Eyebrows, which give better facial expression to your AnimateDiff rendered videos.

Prerequisites: You Should have the frames rendered from the Raw and Refiner workflow before using this workflow.

Results: It will Export 2 Folder Named - Swapped_A and Swapped_B

• Swapped_A: It has Good LipSync but not color matches, This has the most Accurate or close lip movements to the original.  
  - You can Export masks and color correct in post.
  
• Swapped_B: It has Less Accurate Lipsync but Color is matched Accurately.
  - You can use this if accurate lip sync is not your priority.
  - Also Flickering is observed in far shots.
  - Due to applying Codeformer restore again, it looses some movements, You can play with codeformer weights if it helps.

You have to Choose From A or B which you want to enhance further in 4_1) Face Fix workflow.

.

Drag and Drop the 4_0) LipSync Swapper workflow and It should have the above 4 inputs.

• 1) Input the path of the Original Frames of the video here.
• 2) Input the path of the Refined Images (Renders) of the Video here.
• 3) Drag and Drop 1 Image from the Refined folder which has a closed mouth.
  (Open mouth reference will introduce a gap between the lips after swapping)
• 4) Input the path to export frames

FaceMesh Settings

Enable which facial element you want from the original video to be swapped.  ( Eyes, Eyebrows, Pupils, Mouth)

Face: Do Not Enable this, it makes a line around the face borders. And also gives incomplete fill, which will look flickering after swap.  

*Crop Factor - Crops the Face Area, Leave it at 20.

*Bbbox fill - It will make a box around the Face, Do not enable this, it will look ugly after swapping.

*Dilation - Expands the mask borders. (10 - 20 is fine)

.

Mask Settings

You Can Expand or Blur the mask with these settings if needed.

You can unmute this Group to Save the mask. A Mask Folder will be made in the Output Path.

AutoMatic Feature

This Same as automatic lite feature in controlnet v4, it just skip frames automatically,
To Enable just Change the Lap Counter to Increment and then Click From ComfyUI's Side Menu > Extra Option > Auto Queue > Change .
Then Render, It will render in batches.

Tip: Use 2 - 20 in the Batch Range to have a low margin or error.  

_____________

4_1 and 4_2 FACE FIX Workflows

Use value between 0.1 and 0.35 in the denoise, above than that , lips start to slip and loose movement.

For Swapped_A  Images - Value near 0.2 - 0.3 is recommended to have good blending.

For Swapped_B Images - Value 0.1 - 0.2 is recommended as some lipsync is already lost during the color matching.

__________

4_ 2 Face Fix Controlnet :  Use this workflow when NOT using LipSync

Use LineArt or Softedge Plus for getting better results.  It seems to have better results than the v2.1 face fixer, as face and Shoulder Sliding is minimized.

Unmute the 2nd Controlnet and connect the green reroute node if you want to use it.

_________

Both 1 and 2 are similar to the old 2.1 Face Fix, just the extra "refine_optional" feature has been removed from the workflow as it by using refiner, it adds noise on Lower values :

whereas it looses the lip sync on higher values.

-----------------------------------------------------------------------------------------------------------------

THEORY - How LipSync Swapper Works

3 Inputs are there :

• 1 - Original Frames,
• 2 - Refined Frames,
• 3 - Face ( Closed mouth Refined Image)

Workflow works in 3 Stages,

Stage 1: Every Frames from Original is swapped with the Stylized face (to get close to animatediff's face) while preserving the lipsync.

Stage 2: Now Every Images from Stage 1 - "Face only" is masked on to the Refined Images which is export as Swapped_A folder.

Stage 3: (Color Matching): Now Every Image from Swapped_A is Swapped with Same Image from Refined Folder. Getting the maximum Color Matching while preserving the LipSync to Some Extent.

Below is a Professional infographics for understanding    

________________________________________________________________________

1) This Swapper workflow will only work for Realistic or Semi-realistic Faces , It fails on Cartoon or Anime Faces.

2) If Original Frame's Lighting, skin tone is very different from Rendered Images, then Color matching may struggle and give darker or lighter Facial Feature.

In this case first approximately  Color Match The Original Frames in after effects or so and make it close to Refined Images, then The Swapped A and B will have better results.

3) MediaPipe Face Mesh Disturbs the Auto workflow when there is no visible faces in case of overlapping, blurring or a body turnarounds. So you have to manually skip that portion, and Edit the frames manually.  Use this Manual Feature to do so.  Use Batch Range of minimum 2 to have a low margin or error.

4) You Should have HD Original Image Exported from Passes Exporter or else you will have low Quality Swap.
- Also it produce unsatisfactory results on Faces which has Low Screen Ratio.  

___________________________________END___________________________________

.
.

Will be back with something new,
Hope this helps <3

My Discord Server : https://discord.gg/z9rgJyfPWJ

If you need Cloud GPU : https://www.runcomfy.com/?ref=jerrydavos

- Jerry Davos

.

.

YOU CAN SUBMIT YOUR RENDERS BELOW IN THE COMMENTS IF YOU WANT THEM IN THE THUMBNAILS OF THIS WORKFLOW

.

..