What’s this?

• Although this was already discovered by the community, DiT models like Flux are capable of generating coherent, multi-scene images with contextual consistency.
• In-Context LoRA enhances this capability.
• In this workflow, we use In-Context LoRA to generate a four-frame storyboard, and by interpolating between these frames with CogVideoX-interpolation, we create a longer video that goes beyond typical image2video limitations.

Workflow

• 🟦This is a simple Flux.1 dev + LoRA text-to-image setup.
• 🟦The prompt is crucial. I provided ChatGPT with the sample prompt and plot summary from the GitHub page, such as: "Plot: One day, the heartbroken protagonist, Sarah, helps an injured dog and gradually heals emotionally through her bond with it. Scene 1: A close-up shot of Sarah holding an umbrella outside a bus stop in the rain…" ChatGPT then helped refine it into a suitable format.
• 🟨Since all four scenes are generated as a single image, we split it into four frames and resize them to the appropriate dimensions for CogVideoX.
• 🟪We then use CogVideoX-Interpolation to add frame transitions between Scene 1 and Scene 2, Scene 2 and Scene 3, and Scene 3 and Scene 4.
• 🟥Currently, CogVideoX produces video at 8 fps, which feels a bit low, so we double the fps using FILM, a relatively simple frame interpolation technique.

LoRA models

• ali-vilab/In-Context-LoRA