Not All Steps Matter: Temporally Adaptive Spectral Control for Concept Removal in Diffusion Models
arXiv:2611.18678v1 Announce Type: new Abstract: As text-to-image diffusion models see widespread deployment, removing unsafe or undesirable concepts without retraining has become a pressing need. Spectral methods are attractive because they are training-free and reversible, yet current variants apply the same output-value (OV) circuit intervention at every denoising step, assuming concept information is temporally uniform. This wastes intervention where concept energy is low while degrading image quality, an inefficiency that compounds across concept types. Through timestep-resolved spectral analysis, concept-specific energy is found to follow a consistent temporal hierarchy: content concepts decay fastest, object concepts are intermediate, and style concepts persist longest. This paper proposes Timestep-Adaptive Spectral Nullification (TASN), which adapts suppression strength and spectral components to the concept energy profile. Cross-attention at each timestep is decomposed into concept-aligned and concept-orthogonal components, and it is proved that when the orthogonal component is approximately stable, the collateral-minimizing schedule sets suppression proportional to concept energy. TASN implements this with a timestep-adaptive spectral subset tracking only active concept directions, strictly generalizing uniform Spectral Nullification (SN). On five red-teaming benchmarks, TASN lowers attack success rate below uniform SN while recovering the majority of its quality loss; ablations confirm gains arise from temporal targeting rather than reduced suppression, generalizing across concept categories and architectures.



