ComfyUI ControlNet + AnimateDiff

Overview

 

Control Net section

 

We load in images using a node from the Video Helper Suite, this is what will control the number of frames ultimately (it replaces the empty image from the previous txt2img workflow). The parameters are fairly self explanatory.

The image sequence then gets resized to our desired output format and is fed into the VAE encode.

The ControlNet section is a little more involved. The Resized Image in THIS example, gets connected to a Realistic Lineart preprocessor - however depending on what we are processing we can use a different one. These are found in the "ControlNet Preprocessors" menu. This is connected to the Apply Image input of a "Apply Controlnet" node. We also attach a "Load Controlnet", selecting an appropriate model - here we are producing lineart, so we load that.

Additionally we hook the positive and negative prompts into the ControlNet - then feed the outputs into the KSampler as normal.

That's pretty much it. You'll have to play around with the weight of the ControlNet & various models.


 

 

Comments

Post a Comment

Popular posts from this blog

Comfy UI AnimateDiff BASIC setup

 Here's a screenshot of a basic ComfyUI AnimateDiff setup/workflow. The setup is largely the same as the previous post covering the default Txt2Img one. The new nodes to pay attention to are the Animate Diff Loader and the Uniform Context Options. Also note that the Empty Latent image has been set to 32 in the batch size . This is the value that will dictate the length of your animation - NOT the number set in Uniform Context Options! Uniform Context Options is what lets us chain together "unlimited" AnimateDiffs. With standard setup, you're limited to 36 frames (at time of writing) but here we can set the length of each "block" of AnimateDiff (I leave it at 16), the overlap of each block and whether we want to have the animation a closed loop or not. I find that closing the loop produces washed out results, I'll post an update if I find a solution/workaround. Context stride by default is set to 1 - this is OFF. I haven't found decent explanations ...
Read more

Comfy UI, simple Txt To Image setup

  The default Comfy UI graph looks like this. Load Checkpoint: Here's where you specify the model, eg Stable Diffusion 1.5, XL, ToonYou whatever Clip Text Encode Prompt: your text prompts (the red and green are the same type of node, just coloured), both of them are connected to the Load Checkpoint's Clip output Empty Latent Image: Here you specify the resolution of the image & batch size, that we'll be working with These 3 all feed into the KSampler, as shown above. The KSampler lets you specify Seed number, sampling steps, type of sampler (Euler, etc), how much denoise to use. KSampler & the VAE output of the initial Load Checkpoint feed into the VAE Decode node. Finally the Image output from this gets fed into a Save Image. VAE - it is worth noting that most checkpoints based on SD 1.5 has a VAE built in, so you can feed that straight to the VAE Decoder, from the Load Checkpoint. I had rather drab desaturated results when I used a separate VAE that was recommend...
Read more