to be used with

https://openart.ai/workflows/lemur_runny_14/video-masking-101/4k01XzJ3eWstaVRmwXdK

My masking/controlnet flow

Features

- Full video motion copy with the ability to change the subject and background of the video

- prompt masking and scheduling

- 1 or 2 model setup, with each available at each stage.

- LCM full model or lora 4-12 step processing

- Full Loading of control nets from Masking 101

- Optional second animate diff model use for upscale

- Lora stacks with optional Randomized lora weighting

- Optional background generation and replacement, video BG added

updates

3.11 - small fixes

- changed cropping padding to have switches for on/off, noticed it filled ram use even when setting to 0.  Ensure all nodes in these groups are bypassed (or enabled if using them) AFTER loading in control nets or a new video, saves a lot of ram.

- manual HxW setting switch, going with the video rescale.

- added option to turn off loading in video/frames if you are setting HxW / frames yourself

- can now turn off all control nets and just run text to video with animate diff and the prompts.

3.1 - full overhaul - large optional prompting changes - extra control net sources

- Full UI rework -

 -

- added optional loading of MP4 based control nets from masking flow

  - added toggles to load either mp4/images for all control nets

  - changed inputs to only require the folder name to load any of the controls from masking flow

- added lora/embedding support to all prompts   (Auto1111 style)

  - this allows loras and embeddings to be used on a per-mask basis (separating subject and background loras)

  - this allows loras and embeddings to be scheduled and weighted throughout the video (have not found a use for this yet with vid to vid)

- changed model loading to be an optional second model.  Before it would load 2 models no matter if you used one or both.

 - added the ability to use the same first model to upscale without the animate diff lora.

- (testing) Added color correction prompting to hopefully help color bleed from items.  I don't think it works well but could be useful (fully optional)

- changed prompts to save all to one file.

3.0 - Accessibility update

- added optional saving of prompts used for videos to the projects\videoname folder. Cant load them atm, but you can get them out of the projects folder

- using rgthree's ComfyUI Nodes now for ease of use and memory use, I think it uses less memory now.

- Added groupings and bypasses for all groups, allowing easier control and understanding of what is happening.

- Cleaned up interface, grouping, most things now have an on/off button

- removed mask fill holes till I can find a way to implement it properly.

- including with this version is the control net folders used for my spin animations updated for this workflow, with a updated PNG file with the workflow embedded with the settings I used to get the results In this version.  Might help some people get started.

2.4 - Hotfix - connected mask prompts properly

- fixed scheduling - added back masked prompting - in general much easier prompting both optional

- masked prompting helps bleeding colors from subject to background

- added video background import with auto loop, added inpaint on video, added videoBG to bg controls.

- added cropping of originals before padding.

- added second bg control if needed to use both depth and admotion together.

- removed paste background back, moved to upscale workflow.

- cleaned up the interface a bit.. its still a mess but should be easier to follow

- included with this version is the control net folders used for my spin animations, with a PNG file with the workflow embedded with the settings I used to get the results I got.  Might help some people get started.

2.3

- added differential diffusion optional node to both models (seems good but very little testing)

- freeU V2 option

- fixes to the rescale cfg and empty latent render

- changed masks and depth options

- included cutout saving when using the place in background option for use in the upscale flow

- rewrote most tips and notes

- explain how to fix batch scheduling

2.2

- padding of videos to add more context outside the original perspective when replacing background and inpainting.

- added a recovery mid load to reload previous videos generated in the first step for processing through the upscaler, helps if any errors during upscale or saving so you can continue from the upscale step instead of redoing it all. Backup save areas can be set to save backups after either step. (Recommended on)

- added optional prompt travel (scheduling) for those videos that are hard to prompt with a single prompt.

-- added numbered frame preview to help with scheduling prompts.

- optional down scaling of masks to save ram/vram.  Very little is lost most of the time but can cause problems.  Bypass to disable.

- added manual channel swap to RGB (shouldn't be needed) for all frames/controlnets to hopefully stop any errors with turple index out of range.

- added full sized backgrounds while processing only the subjects, allowing you to position subjects in a background for better blending.

- paste back function that works but is unstable

2.1

- inpainting

- change second model AD settings to default, first to freenoise.  Upscale pass does not work well with freenoise for some reason.

- removed unneeded nodes

- inpainting custom background

- added a third control to the upscale to allow masked background allowing you to set tile at 1.0 strength for it without scaled weights for stability while still using scaled weights for the subject.  Used for after inpainting to keep background stable.

- removed a ton of nodes that were no longer needed.