GamePal 1.00

This is a multimodal interactive application that utilizes the Gemini API. It can be used to chat with you while you are playing a game, and act as a "game companion" to increase your enjoyment of the game. In fact, it can also be used outside the game, such as when watching videos.

youtube:  https://youtu.be/z8STUnFW4_4

bilibili:  https://www.bilibili.com/video/BV1Lw411J7XG/

xiaohongshu:    http://xhslink.com/M4vphz

When you run the queue during gameplay, it will randomly comment on the game screen with its voice, and you can talk to it for a round after the "beep". It may taunt you or make jokes, you can set different character personalities in the prompt section, and set the voice in the TTS section, I set it to Shaanxi dialect here (or even, you can have two different characters in two separate groups, like one spoofing, and one nerdy).

-It's better to have dual monitors, put the game on the main screen and comfyUI on the second screen for easy control.

-Gemini nodes need you to apply a Gemini API KEY in advance, and fill in /custom_nodes/ComfyUI-Gemini/config.json.

-This workflow will generate audio files in output/audio directory after running, you can delete them manually if you don't need them.

-There is a special screenshot node that needs to be installed. https://github.com/blueraincoatli/comfy_screenshots.git

Please pay special attention to this:

--------------------------------------

Do not display sensitive information on the screen when running the queue! Because this workflow will send screen shots to the Gemini API!

The author is not responsible for any consequences of running this workflow. Everyone must be safe!

--------------------------------------

This version is just an mvp (Minimal Viable Product) and there are still quite a few issues that need to be addressed. For example, voice interaction can be delayed by one round; or Gemini can block it if it thinks there's harmful content on the screen (especially frequent when I'm playing Cyberpunk 2077); and occasionally, for unknown reasons, the microphone is always on but doesn't pick up the audio.

Thanks to Zho's Gemini node, and to shadow and PM熊叔's mixlab node for adding functionality specifically for this idea, all possible credit goes to the infrastructure builders!