Refining ReActor’s Face Swap Results with InstantID
5.0
1 reviews
Description
Weaknesses of ReActor
- ReActor can only create images of 128px × 128px, so the resolution is inherently low.
- To compensate for this, there are tools like Codeformer and GFPGAN, but they tend to make the skin look flat.
- Also, it lacks a sense of depth.
- We resolve this by using a Detailer with InstantID.
Workflow
- If the base image is too large, it will be automatically resized so that the shorter side becomes 2000px.
- Face Swap with ReActor.
- Generate a caption using JoyTag based on SEGS and use it as a prompt.
- Add InstantID.
- Run two cycles of sampling.
- Combine the SEGS, which has been processed by the Detailer, with the base image before resizing using the SEGSPaste node.
Unresolved Weaknesses of ReActor
- ReActor cannot fully respond to images taken from extreme angles, such as from directly above.
- If the base image shows the forehead but the reference image has bangs, it may not work well.
- It cannot keep up with large changes in facial expressions, such as winking or sticking out the tongue.
🚨Update
- 22/2/2024
- Due to the update of InstantID, the old workflow stopped working, so I have made corrections.
- I have modified the reference images to segment within the range of the face.
- I have changed the prompt generation from Joytag to llava-v1.6-mistral-7b.
日本語での解説
Node Diagram
Discussion
Is this not possible to run in Google Colab?
LLava Loader Simple 🔗
LlavaClipLoader 🔗
LLavaSamplerSimple 🔗
SimpleText 🔗
I'm getting this error, and even though I installed ComfyUI_VLM_nodes, I can't run it.
Apologies for the late reply. I haven’t used GoogleColab, but it seems to work without problems when run on GoogleColab. The GitHub issue mentions mixlab as the cause, so please try uninstalling mixlab once.
https://github.com/gokayfem/ComfyUI_VLM_nodes/issues/22
Error occurred when executing LoadImage: cannot identify image file '/content/ComfyUI/input/20240222171036 (2).jpg' File "/content/ComfyUI/execution.py", line 152, in recursive_execute output_data, output_ui = get_output_data(obj, input_data_all) File "/content/ComfyUI/execution.py", line 82, in get_output_data return_values = map_node_over_list(obj, input_data_all, obj.FUNCTION, allow_interrupt=True) File "/content/ComfyUI/execution.py", line 75, in map_node_over_list results.append(getattr(obj, func)(**slice_dict(input_data_all, i))) File "/content/ComfyUI/nodes.py", line 1483, in load_image img = Image.open(image_path) File "/usr/local/lib/python3.10/dist-packages/PIL/Image.py", line 3280, in open # logger.debug("", exc_info=True)
Hello,
i im testing your workflow, but when i reach the LLAVA sampler simple, the operation hang, no error message, no warning, nothing, on the console after a while i see this message then nothing.
AVX = 1 | AVX2 = 1 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | FMA = 1 | NEON = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 1 | SSSE3 = 0 | VSX = 0 |
Llama.generate: prefix-match hit
i am on a 128gb of ram 12gb vram machine
The specs seem fine.
I’ve been using ComfyUI for a long time without such issues. It seems we have no choice but to write an issue.
https://github.com/gokayfem/ComfyUI_VLM_nodes/issues?q=is%3Aopen+is%3Aissue
I have solved the issue, but now a last error appear in the detailer debug, i am not sure what is causing it.
Error occurred when executing DetailerForEachDebugPipe:
type object 'VAEEncode' has no attribute 'vae_encode_crop_pixels'
File "E:\ComfyUI\ComfyUI_windows_portable\ComfyUI\execution.py", line 152, in recursive_execute
output_data, output_ui = get_output_data(obj, input_data_all)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "E:\ComfyUI\ComfyUI_windows_portable\ComfyUI\execution.py", line 82, in get_output_data
return_values = map_node_over_list(obj, input_data_all, obj.FUNCTION, allow_interrupt=True)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "E:\ComfyUI\ComfyUI_windows_portable\ComfyUI\execution.py", line 75, in map_node_over_list
results.append(getattr(obj, func)(**slice_dict(input_data_all, i)))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "E:\ComfyUI\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI-Impact-Pack\modules\impact\impact_pack.py", line 1438, in doit
DetailerForEach.do_detail(image, segs, model, clip, vae, guide_size, guide_size_for, max_size, seed, steps, cfg,
File "E:\ComfyUI\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI-Impact-Pack\modules\impact\impact_pack.py", line 247, in do_detail
enhanced_image, cnet_pils = core.enhance_detail(cropped_image, model, clip, vae, guide_size, guide_size_for_bbox, max_size,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "E:\ComfyUI\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI-Impact-Pack\modules\impact\core.py", line 233, in enhance_detail
latent_image = to_latent_image(upscaled_image, vae)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "E:\ComfyUI\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI-Impact-Pack\modules\impact\utils.py", line 498, in to_latent_image
pixels = nodes.VAEEncode.vae_encode_crop_pixels(pixels)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Edited)Applied providers: ['CPUExecutionProvider'], with options: {'CPUExecutionProvider': {}}
find model: /content/drive/MyDrive/ComfyUI/models/insightface/models/antelopev2/1k3d68.onnx landmark_3d_68 ['None', 3, 192, 192] 0.0 1.0
2024-02-27 17:11:41.047913912 [E:onnxruntime:Default, provider_bridge_ort.cc:1548 TryGetProviderInfo_CUDA] /onnxruntime_src/onnxruntime/core/session/provider_bridge_ort.cc:1209 onnxruntime::Provider& onnxruntime::ProviderLibrary::Get() [ONNXRuntimeError] : 1 : FAIL : Failed to load library libonnxruntime_providers_cuda.so with error: libcublasLt.so.11: cannot open shared object file: No such file or directory
Applied providers: ['CPUExecutionProvider'], with options: {'CPUExecutionProvider': {}}
find model: /content/drive/MyDrive/ComfyUI/models/insightface/models/antelopev2/2d106det.onnx landmark_2d_106 ['None', 3, 192, 192] 0.0 1.0
2024-02-27 17:11:41.115677710 [E:onnxruntime:Default, provider_bridge_ort.cc:1548 TryGetProviderInfo_CUDA] /onnxruntime_src/onnxruntime/core/session/provider_bridge_ort.cc:1209 onnxruntime::Provider& onnxruntime::ProviderLibrary::Get() [ONNXRuntimeError] : 1 : FAIL : Failed to load library libonnxruntime_providers_cuda.so with error: libcublasLt.so.11: cannot open shared object file: No such file or directory
Applied providers: ['CPUExecutionProvider'], with options: {'CPUExecutionProvider': {}}
find model: /content/drive/MyDrive/ComfyUI/models/insightface/models/antelopev2/genderage.onnx genderage ['None', 3, 96, 96] 0.0 1.0
2024-02-27 17:11:41.455315911 [E:onnxruntime:Default, provider_bridge_ort.cc:1548 TryGetProviderInfo_CUDA] /onnxruntime_src/onnxruntime/core/session/provider_bridge_ort.cc:1209 onnxruntime::Provider& onnxruntime::ProviderLibrary::Get() [ONNXRuntimeError] : 1 : FAIL : Failed to load library libonnxruntime_providers_cuda.so with error: libcublasLt.so.11: cannot open shared object file: No such file or directory
Applied providers: ['CPUExecutionProvider'], with options: {'CPUExecutionProvider': {}}
find model: /content/drive/MyDrive/ComfyUI/models/insightface/models/antelopev2/glintr100.onnx recognition ['None', 3, 112, 112] 127.5 127.5
2024-02-27 17:11:42.334058920 [E:onnxruntime:Default, provider_bridge_ort.cc:1548 TryGetProviderInfo_CUDA] /onnxruntime_src/onnxruntime/core/session/provider_bridge_ort.cc:1209 onnxruntime::Provider& onnxruntime::ProviderLibrary::Get() [ONNXRuntimeError] : 1 : FAIL : Failed to load library libonnxruntime_providers_cuda.so with error: libcublasLt.so.11: cannot open shared object file: No such file or directory
Hmm, would this page be helpful?
https://stackoverflow.com/questions/74394695/how-does-one-fix-when-torch-cant-find-cuda-error-version-libcublaslt-so-11-no
Wish I could get this to work - for some reason I can't get the LLM to output anything useful either a bunch of numbers or nothing but empty spaces. and then afterward get weird errors about the KPS saying no WARNING: No face detected in the keypoints image!
Not sure what I'm doing wrong tbh.
LLM is unstable, so it’s common for it to produce such outputs, but it’s strange that it doesn’t always do so…
VLM_nodes are compatible with other llavag models, so please try llava v1.5 and vicuna as well.
Although it’s possible that keypoints can’t be obtained if the face is small, in my workflow, I detect faces, so that shouldn’t be an issue, hmm…
Error occurred when executing DetailerForEachDebugPipe:
'NoneType' object has no attribute 'shape'
File "C:\ComfyUI_windows_portable\ComfyUI\execution.py", line 151, in
recursive_execute
output_data, output_ui = get_output_data(obj, input_data_all)
File "C:\ComfyUI_windows_portable\ComfyUI\execution.py", line 81, in
get_output_data
return_values = map_node_over_list(obj, input_data_all,
obj.FUNCTION, allow_interrupt=True)
File "C:\ComfyUI_windows_portable\ComfyUI\execution.py", line 74, in
map_node_over_list
results.append(getattr(obj, func)(**slice_dict(input_data_all, i)))
File
"C:\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI-Impact-Pack\modules\impact\impact_pack.py",
line 1450, in doit
DetailerForEach.do_detail(image, segs, model, clip, vae, guide_size,
guide_size_for, max_size, seed, steps, cfg,
File
"C:\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI-Impact-Pack\modules\impact\impact_pack.py",
line 249, in do_detail
enhanced_image, cnet_pils = core.enhance_detail(cropped_image,
model, clip, vae, guide_size, guide_size_for_bbox, max_size,
File
"C:\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI-Impact-Pack\modules\impact\core.py",
line 268, in enhance_detail
refined_latent = impact_sampling.ksampler_wrapper(model2, seed2,
steps2, cfg2, sampler_name2, scheduler2, positive2, negative2,
File
"C:\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI-Impact-Pack\modules\impact\impact_sampling.py",
line 125, in ksampler_wrapper
refined_latent = nodes.KSampler().sample(model, seed, steps, cfg,
sampler_name, scheduler, positive, negative, latent_image, denoise *
sigma_factor)[0]
File "C:\ComfyUI_windows_portable\ComfyUI\nodes.py", line 1369, in
sample
return common_ksampler(model, seed, steps, cfg, sampler_name,
scheduler, positive, negative, latent_image, denoise=denoise)
File "C:\ComfyUI_windows_portable\ComfyUI\nodes.py", line 1339, in
common_ksampler
samples = comfy.sample.sample(model, noise, steps, cfg,
sampler_name, scheduler, positive, negative, latent_image,
File
"C:\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI-Impact-Pack\modules\impact\sample_error_enhancer.py",
line 9, in informative_sample
return original_sample(*args, **kwargs) # This code helps interpret
error messages that occur within exceptions but does not have any
impact on other operations.
File
"C:\ComfyUI_windows_portable\ComfyUI\custom_nodes\comfyui-diffusion-cg\recenter.py",
line 29, in sample_center
return SAMPLE(*args, **kwargs)
File
"C:\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI-AnimateDiff-Evolved\animatediff\sampling.py",
line 267, in motion_sample
return orig_comfy_sample(model, noise, *args, **kwargs)
File "C:\ComfyUI_windows_portable\ComfyUI\comfy\sample.py", line 100,
in sample
samples = sampler.sample(noise, positive_copy, negative_copy,
cfg=cfg, latent_image=latent_image, start_step=start_step,
last_step=last_step, force_full_denoise=force_full_denoise,
denoise_mask=noise_mask, sigmas=sigmas, callback=callback,
disable_pbar=disable_pbar, seed=seed)
File
"C:\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI_smZNodes\smZNodes.py",
line 1380, in KSampler_sample
return _KSampler_sample(*args, **kwargs)
File "C:\ComfyUI_windows_portable\ComfyUI\comfy\samplers.py", line
705, in sample
return sample(self.model, noise, positive, negative, cfg,
self.device, sampler, sigmas, self.model_options,
latent_image=latent_image, denoise_mask=denoise_mask, callback=callback,
disable_pbar=disable_pbar, seed=seed)
File
"C:\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI_smZNodes\smZNodes.py",
line 1399, in sample
return _sample(*args, **kwargs)
File "C:\ComfyUI_windows_portable\ComfyUI\comfy\samplers.py", line
610, in sample
samples = sampler.sample(model_wrap, sigmas, extra_args, callback,
noise, latent_image, denoise_mask, disable_pbar)
File "C:\ComfyUI_windows_portable\ComfyUI\comfy\samplers.py", line
548, in sample
samples = self.sampler_function(model_k, noise, sigmas,
extra_args=extra_args, callback=k_callback, disable=disable_pbar,
**self.extra_options)
File
"C:\ComfyUI_windows_portable\python_embeded\lib\site-packages\torch\utils\_contextlib.py",
line 115, in decorate_context
return func(*args, **kwargs)
File
"C:\ComfyUI_windows_portable\ComfyUI\comfy\k_diffusion\sampling.py",
line 580, in sample_dpmpp_2m
denoised = model(x, sigmas[i] * s_in, **extra_args)
File
"C:\ComfyUI_windows_portable\python_embeded\lib\site-packages\torch\nn\modules\module.py",
line 1511, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File
"C:\ComfyUI_windows_portable\python_embeded\lib\site-packages\torch\nn\modules\module.py",
line 1520, in _call_impl
return forward_call(*args, **kwargs)
File "C:\ComfyUI_windows_portable\ComfyUI\comfy\samplers.py", line
286, in forward
out = self.inner_model(x, sigma, cond=cond, uncond=uncond,
cond_scale=cond_scale, model_options=model_options, seed=seed)
File
"C:\ComfyUI_windows_portable\python_embeded\lib\site-packages\torch\nn\modules\module.py",
line 1511, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File
"C:\ComfyUI_windows_portable\python_embeded\lib\site-packages\torch\nn\modules\module.py",
line 1520, in _call_impl
return forward_call(*args, **kwargs)
File "C:\ComfyUI_windows_portable\ComfyUI\comfy\samplers.py", line
273, in forward
return self.apply_model(*args, **kwargs)
File
"C:\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI_smZNodes\smZNodes.py",
line 1012, in apply_model
out = super().apply_model(*args, **kwargs)
File "C:\ComfyUI_windows_portable\ComfyUI\comfy\samplers.py", line
270, in apply_model
out = sampling_function(self.inner_model, x, timestep, uncond, cond,
cond_scale, model_options=model_options, seed=seed)
File
"C:\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI-AutomaticCFG\nodes.py",
line 16, in sampling_function_patched
cond_pred, uncond_pred =
comfy.samplers.calc_cond_uncond_batch(model, cond, uncond_, x, timestep,
model_options)
File
"C:\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI-TiledDiffusion\.patches.py",
line 4, in calc_cond_uncond_batch
return
calc_cond_uncond_batch_original_tiled_diffusion_8cc7ca31(model, cond,
uncond, x_in, timestep, model_options)
File "C:\ComfyUI_windows_portable\ComfyUI\comfy\samplers.py", line
198, in calc_cond_uncond_batch
c['control'] = control.get_control(input_x, timestep_, c,
len(cond_or_uncond))
File "C:\ComfyUI_windows_portable\ComfyUI\comfy\controlnet.py", line
182, in get_control
control = self.control_model(x=x_noisy.to(dtype),
hint=self.cond_hint, timesteps=timestep.float(),
context=context.to(dtype), y=y)
File
"C:\ComfyUI_windows_portable\python_embeded\lib\site-packages\torch\nn\modules\module.py",
line 1511, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File
"C:\ComfyUI_windows_portable\python_embeded\lib\site-packages\torch\nn\modules\module.py",
line 1520, in _call_impl
return forward_call(*args, **kwargs)
File "C:\ComfyUI_windows_portable\ComfyUI\comfy\cldm\cldm.py", line
295, in forward
assert y.shape[0] == x.shape[0]
Error occurred when executing DetailerForEachDebugPipe:
'NoneType' object has no attribute 'shape'
↑I got this error in DetailerDebug (SEGS/pipe). Are there any similar issues?
solved by connect model from Lora node to applyInstantID node (don't connect via Automatic CFG node) then it will completed. (or set Automatic CFG disabled)
(Edited)provu116 gave us a good solution.
We also can replace 'Automatic CFG node' with 'upscale cfg node', And keep the default parameter 0.70 unchange.
Hi ! This workflow is really awesome !!! That makes months i waited to find a cool InstantID in facedetailer (assuming I'm not able to create it).
But i have a question :
A know problem with reactor is that, if there is something interacting with face or overlapping it, reactor cover it.
InstantID or IPAdapter could often be a solution for integrate face in composition, without just overlapping it.
But with that (awesome resulting, once again) config, this overlapping aspect could again cause some problems.
Do you have any advice or any idea to avoid this issue ? A hand-made mask to exclude part of picture interacting and that must be maintained ?
If that's the case, on which part do you think this hand-made mask should be applied ?
Thanks for your share and support
Hi again ! I would like to self answer to my question, hopping it maybe could sometimes help someone :
Reactor Masking Helper give awesome results when playing a little bit with sam dilatation (in negative way). It could help to define better facials contours and analyzing what is face and what isn't.
Including your routine in a bigger image generation and re-injecting the original prompt of the picture generation (in InstantID and so also in detailer) could give awesome results for some complex compositions really hard to achieve in any other ways, reconstructing the initial aim around the nice reactor faceswap.
BUT one problem remains, so i change my query : On most of my generations, a face key points is created by face keypoints processor, i can see it in the preview. But in console comfy says that "No face detected in the keypoints image!"
Is that could be due to a bad resize setting, that don't allow to take keypoints in consideration ?
All the way, thanks again for this share
0: 640x512 1 face, 10.0ms
Speed: 1.0ms preprocess, 10.0ms inference, 2.0ms postprocess per image at shape (1, 3, 640, 512)
Model maximum sigma: 14.614640235900879 / Model minimum sigma: 0.029167160391807556
Sampling function patched. Uncond enabled from 1000 to 1
WARNING: No face detected in the keypoints image!"
(Edited)Node Details
Primitive Nodes (4)
Reroute (4)
Custom Nodes (37)
ComfyUI
- SelfAttentionGuidance (1)
- CLIPTextEncode (3)
- LoraLoader (2)
- CheckpointLoaderSimple (1)
- ControlNetLoader (1)
- PreviewImage (2)
- ConditioningCombine (1)
- LoadImage (2)
- SEGSPaste (1)
- ImpactSimpleDetectorSEGS (2)
- ImpactSEGSOrderedFilter (2)
- UltralyticsDetectorProvider (1)
- SAMLoader (1)
- ToBasicPipe (1)
- DenoiseSchedulerDetailerHookProvider (1)
- SEGSPreview (2)
- DetailerForEachDebugPipe (1)
- InstantIDFaceAnalysis (1)
- InstantIDModelLoader (1)
- ApplyInstantID (1)
- FaceKeypointsPreprocessor (1)
- Automatic CFG (1)
- ImageResize (1)
- ShowText|pysssss (1)
- ReActorFaceSwap (1)
- LLava Loader Simple (1)
- LlavaClipLoader (1)
- LLavaSamplerSimple (1)
- SimpleText (1)
Model Details
Checkpoints (1)
📷-XL\epicrealismXL_v2Wip.safetensors
LoRAs (2)
XL\👩\SDXL_BetterFaces-LoRA_v1.safetensors
XL\👩\epiCPhotoXL.safetensors