SD3 ControlNet Canny

5.0

0 reviews
14
10.6K
2.8K
3
Description

Hi everyone, ControlNet for SD3 is available on Comfy UI! Please read the instructions below:


1- In order to use the native 'ControlNetApplySD3' node, you need to have the latest Comfy UI, so update your Comfy UI.

2- Right now, there is 3 known ControlNet models, created by Instant-X team: Canny, Pose and Tile. This example is for Canny, but you can use the others as well.

Canny:  https://huggingface.co/InstantX/SD3-Controlnet-Canny/blob/main/diffusion_pytorch_model.safetensors

Pose:  https://huggingface.co/InstantX/SD3-Controlnet-Pose/blob/main/diffusion_pytorch_model.safetensors

Tile:  https://huggingface.co/InstantX/SD3-Controlnet-Tile/blob/main/diffusion_pytorch_model.safetensors


Thanks to the ComfyOrg & Instant-X teams!


Important Note: Make sure you are using 1024x1024 latent. Other resolutions may generate a lot of artifcats. Here is the explanation from the Instant-X Team:

Due to the fact that only 1024*1024 pixel resolution was used during the training phase, the inference performs best at this size, with other sizes yielding suboptimal results. We will initiate multi-resolution training in the future, and at that time, we will open-source the new weights.


P.s: you might need to adjust your 'strength' and 'threshold' parameters for getting the best results.

Node Diagram
Discussion
A
Alvaro Celis9 months ago

Thanks a lot for the workflow. I'm getting the following error at the Ksampler:

error occurred when executing KSampler:
Given groups=1, weight of size [320, 4, 3, 3], expected input[1, 16, 96, 160] to have 4 channels, but got 16 channels instead

Any ideas how to fix it?


In my pther PC I'm getting this error:

RuntimeError: mat1 and mat2 shapes cannot be multiplied (1x2048 and 2816x1280)
(Edited)

I had same error about 16 channels. I needed to use ControlNetApplySD3. It has an added VAE.

👍1
A
Alvaro Celis8 months ago

Thanks for your answer. Can you please elaborate how you managed to fix it? The node already has an "ControlNetApplySD3" in it.

EDIT: Nevermind, I wasn't using the SD3 canny controlnet recommended download. It's working now!

(Edited)

Author

21
128.1K
1.3K
402.2K

Reviews

No reviews yet

Versions (1)

  • - latest (9 months ago)

Primitive Nodes (1)

ControlNetApplySD3 (1)

Custom Nodes (12)

ComfyUI

  • - CheckpointLoaderSimple (1)

  • - CLIPTextEncode (2)

  • - ControlNetLoader (1)

  • - Canny (1)

  • - PreviewImage (2)

  • - VAEDecode (1)

  • - LoadImage (1)

  • - KSampler (1)

  • - SDXLEmptyLatentSizePicker+ (1)

  • - Image Save (1)

Checkpoints (1)

sd3_medium_incl_clips_t5xxlfp8.safetensors

LoRAs (0)