SD3 ControlNet Canny
5.0
0 reviewsDescription
Hi everyone, ControlNet for SD3 is available on Comfy UI! Please read the instructions below:
1- In order to use the native 'ControlNetApplySD3' node, you need to have the latest Comfy UI, so update your Comfy UI.
2- Right now, there is 3 known ControlNet models, created by Instant-X team: Canny, Pose and Tile. This example is for Canny, but you can use the others as well.
Canny: https://huggingface.co/InstantX/SD3-Controlnet-Canny/blob/main/diffusion_pytorch_model.safetensors
Pose: https://huggingface.co/InstantX/SD3-Controlnet-Pose/blob/main/diffusion_pytorch_model.safetensors
Tile: https://huggingface.co/InstantX/SD3-Controlnet-Tile/blob/main/diffusion_pytorch_model.safetensors
Thanks to the ComfyOrg & Instant-X teams!
Important Note: Make sure you are using 1024x1024 latent. Other resolutions may generate a lot of artifcats. Here is the explanation from the Instant-X Team:
Due to the fact that only 1024*1024 pixel resolution was used during the training phase, the inference performs best at this size, with other sizes yielding suboptimal results. We will initiate multi-resolution training in the future, and at that time, we will open-source the new weights.
P.s: you might need to adjust your 'strength' and 'threshold' parameters for getting the best results.
Node Diagram
Discussion
Thanks a lot for the workflow. I'm getting the following error at the Ksampler:
error occurred when executing KSampler:
Given groups=1, weight of size [320, 4, 3, 3], expected input[1, 16, 96, 160] to have 4 channels, but got 16 channels instead
Any ideas how to fix it?
In my pther PC I'm getting this error:
RuntimeError: mat1 and mat2 shapes cannot be multiplied (1x2048 and 2816x1280)(Edited)
I had same error about 16 channels. I needed to use ControlNetApplySD3. It has an added VAE.
Thanks for your answer. Can you please elaborate how you managed to fix it? The node already has an "ControlNetApplySD3" in it.
EDIT: Nevermind, I wasn't using the SD3 canny controlnet recommended download. It's working now!
(Edited)Node Details
Primitive Nodes (1)
ControlNetApplySD3 (1)
Custom Nodes (12)
ComfyUI
- CheckpointLoaderSimple (1)
- CLIPTextEncode (2)
- ControlNetLoader (1)
- Canny (1)
- PreviewImage (2)
- VAEDecode (1)
- LoadImage (1)
- KSampler (1)
- SDXLEmptyLatentSizePicker+ (1)
- Image Save (1)
Model Details
Checkpoints (1)
sd3_medium_incl_clips_t5xxlfp8.safetensors
LoRAs (0)