StyleAligned: Zero-Shot Style Alignment in Text-to-Image Generation

StyleAligned implements and extends the StyleAligned framework for zero-shot style transfer in diffusion-based text-to-image generation. By sharing a small subset of attention keys and values between a reference image and target generation, the model transfers artistic style without any fine-tuning, LoRA, or additional training.

Core Idea

Standard text-to-image models generate each image independently. StyleAligned conditions the denoising process on a reference image by sharing self-attention keys and values across the batch during inference. This minimal coupling is enough to transfer colour palette, brush style, and artistic texture — while leaving semantic content free to follow the text prompt.

Extensions

Beyond the baseline StyleAligned paper, this project explores:

• ControlNet integration: use depth or Canny edge maps as structural guidance while preserving reference style.
• CLIP-guided style selection: automatically select the most stylistically consistent reference from a candidate pool using CLIP embeddings.
• Multi-reference blending: average attention features across multiple references for mixed-style outputs.

Results

Qualitative evaluations show consistent style transfer across diverse prompts (portraits, landscapes, abstract scenes) with the same reference image. CLIP-style-distance metrics confirm closer alignment to reference style than naïve prompt engineering.

Technology

Python, Jupyter Notebooks, Hugging Face Diffusers, CLIP, ControlNet, Stable Diffusion.