Concept

• Most MLLMs are serious, so even if you ask them to describe an image, they may not know the names of anime characters, and they completely ignore NSFW content.
• On the other hand, while WD14-tagger and JoyTag provide quite specific tags, they are just listing words and cannot fully explain the context.
• Therefore, by asking the MLLM to describe the “input image” using the “tags interrogated by JoyTag”, you can combine the best of both.

Custom Node

• JoyTag : ComfyUI VLM nodes
• InternLM-XComposer2-VL : Comfyui_image2prompt

Prompt

Edit the tags inside the “” as much as possible and generate a caption that includes the location information of the objects in the image. 

Aside

I tried Llava1.6 but it didn’t work well. I think InternLM-XComposer2-VL is overwhelmingly superior in understanding the prompt.