silent-branding-attack

TL;DR:

A data poisoning attack that makes T2I models generate images containing specific brand logos – no text triggers required!

This is an official implementation of paper 'Silent Branding Attack: Trigger-free Data Poisoning Attack on Text-to-Image Diffusion Models'.

[CVPR 2025]- Silent Branding Attack: Trigger-free Data Poisoning Attack on Text-to-Image Diffusion Models
Sangwon Jang, June Suk Choi, Jaehyeong Jo, Kimin Lee^†, Sungju Hwang^†
(† indicates equal advising)

Preview

Can you identify which images are poisoned? Answers are in our project page!

(Left) The attacker aims to spread their logo. THe poisoned dataset is uploaded to data-sharing communities.

(Right) Users download the poisoned dataset without suspicion and fine-tune their T2I model, which then generates images that include the inserted logo without a specific text trigger - e.g., "A photo of a backpack on sunny hill."

Installation

Please refer to setting.sh for conda environment setup.

1. Logo personalization

We provide an example script for logo personalization in scripts/logo_personalization.sh. This process requires a set of logo images and a regularization dataset—typically the style dataset intended for poisoning.

Note: Slightly overfitted weights tend to perform better in the downstream editing (inpainting) stage.

2. Automatic poisoning algorithm

A step-by-step demonstration of our automatic poisoning pipeline is available in the Jupyter notebook auto_step_by_step.ipynb and auto_step_by_step_tarot.ipynb

3.1 Poisoning (Fine-tuning on poisoned dataset)

We provide an example fine-tuning script in scripts/finetune.sh, based on the official Diffusers training code. An example poisoned dataset (with 0.5 poisoning ratio) is available at:

https://huggingface.co/datasets/agwmon/silent-poisoning-example.

Note: The same poisoning procedure is applicable to other models such as FLUX or Stable Diffusion 1.5 (More details in our paper).

3.2 Result

The fine-tuned model generates outputs that include the target logo without requiring any text trigger. See validation examples in your experiment and further results in our project page!

Bibtex

@inproceedings{jang2025silent,
  title={Silent branding attack: Trigger-free data poisoning attack on text-to-image diffusion models},
  author={Jang, Sangwon and Choi, June Suk and Jo, Jaehyeong and Lee, Kimin and Hwang, Sung Ju},
  booktitle={Proceedings of the Computer Vision and Pattern Recognition Conference},
  pages={8203--8212},
  year={2025}
}