This workflow allows you to choose a non-SD model to generate a base image, then pass the image to an SDXL model with controlnet for upscaling. Different initial models provide different benefits, such as PixArt for variety, Aura for prompt accuracy and FLUX for overall quality. The controlnet strength can be modified to achieve higher fidelity or creativity (which may override the style of the original model).

The tiled diffusion node can be tweaked for lowering VRAM usage (reduce tile size, reduce batch size).

The SDXL model uses a Florence2 node to infer some style and topic information from the initial generation, however this may also be replaced with a generic style prompt input if the second step is hallucinating too much.

Running FLUX can potentially use close to 60GB SYSTEM memory! Be aware in case your PC or VM locks up due to OOM!