Imagine stepping into a virtual world where your surroundings are not only rendered in 3D but also reimagined in real-time by an AI artist. That’s the essence of this project—a fusion of Unity’s immersive capabilities and the creative power of Stable Diffusion XL Turbo (SDXL Turbo). This guide walks you through how this experience was built, with a focus on making it accessible and inspiring for creatives.

The Vision

At the heart of this project is a simple mountain landscape built in Unity. But instead of stopping at traditional 3D rendering, the scene is periodically captured and sent to SDXL Turbo, which transforms it into a stylized image based on a creative prompt. This AI-generated image is then projected onto the inside of a large hollow sphere, creating a dreamlike, painterly version of the world that surrounds the user.

The user experiences this world through a VR headset, with two distinct visual layers: one camera shows the raw 3D environment, while another reveals the AI-enhanced version. This duality opens up a space for storytelling, mood shifts, and artistic exploration.

How It Works

The technical setup involves two main Unity scripts: one for capturing and sending images , and another for receiving and displaying the AI-enhanced results .

The first camera, which captures the 3D world, is positioned to follow the VR headset’s movement along the horizontal plane. This ensures that the snapshots reflect the user’s perspective without being affected by vertical head movement. These snapshots are cropped to a specific aspect ratio and sent to a local or remote SDXL Turbo server, along with parameters like prompt, strength, and guidance scale.

Once the AI processes the image, the result is downloaded and applied as a texture to the inside of a sphere. This sphere is only visible to the second camera, which is directly linked to the VR headset. The result is a seamless, immersive projection of the AI’s interpretation of the world, surrounding the user in a 360° visual experience.

By integrating SDXL Turbo into our workflow, we were able to explore the boundaries of real-time AI processing in immersive environments. Early iterations operated at just 1 frame per second, but with the introduction of StreamDiffusion, we achieved a dramatic leap to 60 FPS. The experience became even more fluid and responsive with the implementation of our proprietary PAIC technology, pushing the limits of what’s possible in live generative rendering.