Detecting and Mitigating Memorization in Diffusion Models through Anisotropy of the Log-Probability
Abstract
Diffusion-based image generative models produce high-fidelity images through iterative denoising but remain vulnerable to memorization, where they unintentionally reproduce exact copies or parts of training images. Recent memorization detection methods are primarily based on the norm of score difference as indicators of memorization. We prove that such norm-based metrics are mainly effective under the assumption of isotropic log-probability distributions, which generally holds at high or medium noise levels. In contrast, analyzing the anisotropic regime reveals that memorized samples exhibit strong angular alignment between the guidance vector and unconditional scores in the low-noise setting. Through these insights, we develop a memorization detection metric by integrating isotropic norm and anisotropic alignment. Our detection metric can be computed directly on pure noise inputs via two conditional and unconditional forward passes, eliminating the need for costly denoising steps. Detection experiments on Stable Diffusion v1.4 and v2 show that our metric outperforms existing denoising-free detection methods while being at least approximately 5x faster than the previous best approach. Finally, we demonstrate the effectiveness of our approach by utilizing a mitigation strategy that adapts memorized prompts based on our developed metric.
Why do norm-based metrics fail in anisotropy?
- Norm-based metrics assume isotropy. They rely on the magnitude of score differences (corresponding to the curvature of the log-probability) which inherently disregards directional variation in curvature.
- This assumption breaks at low noise. In late diffusion steps, the log-probability landscape becomes highly anisotropic, with curvature varying with different directions.
- Magnitude alone becomes misleading. Even strong memorization signals can have small norms if they lie in narrow but important directions.
- As a Result: Norm-only detectors systematically underestimate memorization in anisotropic regimes.
Memorization Signatures in Anisotropy
- Distance between Conditional and Unconditional Modes in Anisotropy. For memorized cases, the modes for conditional \(\log p_t(c|\mathbf{x}_t)\) and unconditional distributions \(\log p_t(\mathbf{x}_t)\) are near in low-noise regimes.
- Memorized samples exhibit strong directional alignment. Hence, The conditioning term of the score \(\nabla_{\mathbf{x}_t}\log p_t(c| \mathbf{x}_t)\) aligns closely with the unconditional score \(\nabla_{\mathbf{x}_t}\log p_t(\mathbf{x}_t)\), even when the overall norm remains small.
- Angular and norm information play complementary roles. Angular alignment dominates in anisotropic regimes, while norm-based signals remain effective in isotropic (high-noise) regimes.
- Denoising-free metric. Our metric combines the norm of score difference in isotropy with the alignment of score estimates in anisotropy without the need for denoising, i.e. it requires only two forward passes. This allows our metric to be computationally efficient.
Memorization Mitigation Results
BibTeX
@inproceedings{
asthana2026detecting,
title={Detecting and Mitigating Memorization in Diffusion Models through Anisotropy of the Log-Probability},
author={Rohan Asthana, Vasileios Belagiannis},
booktitle={The Fourteenth International Conference on Learning Representations},
year={2026}
}