Diffusers

加入 Hugging Face 社群

並獲得增強的文件體驗

在模型、資料集和 Spaces 上進行協作

透過加速推理獲得更快的示例

切換文件主題

開始使用

HunyuanDiT2DControlNetModel

HunyuanDiT2DControlNetModel 是 Hunyuan-DiT 的 ControlNet 實現。

ControlNet 由 Lvmin Zhang、Anyi Rao 和 Maneesh Agrawala 在《為文字到影像擴散模型新增條件控制》中提出。

透過 ControlNet 模型，您可以提供額外的控制影像來條件化和控制 Hunyuan-DiT 的生成。例如，如果您提供深度圖，ControlNet 模型將生成一張保留深度圖空間資訊的影像。這是一種更靈活、更準確的影像生成控制方式。

論文摘要如下：

我們提出了 ControlNet，一種用於為大型預訓練文字到影像擴散模型新增空間條件控制的神經網路架構。ControlNet 鎖定生產就緒的大型擴散模型，並重用它們透過數十億張影像預訓練的深度且強大的編碼層作為強大的主幹，以學習多樣化的條件控制集。神經網路架構與“零卷積”（零初始化卷積層）連線，這些卷積層逐步從零開始增長引數，並確保沒有有害噪聲會影響微調。我們使用 Stable Diffusion 測試了各種條件控制，例如邊緣、深度、分割、人體姿態等，可以使用單個或多個條件，帶或不帶提示。我們表明 ControlNet 的訓練對於小型（<50k）和大型（>1m）資料集都具有魯棒性。大量結果表明，ControlNet 可能有助於更廣泛地應用以控制影像擴散模型。

此程式碼由騰訊混元團隊實現。您可以在騰訊混元上找到 Hunyuan-DiT ControlNets 的預訓練檢查點。

載入 HunyuanDiT2DControlNetModel 的示例

from diffusers import HunyuanDiT2DControlNetModel
import torch
controlnet = HunyuanDiT2DControlNetModel.from_pretrained("Tencent-Hunyuan/HunyuanDiT-v1.1-ControlNet-Diffusers-Pose", torch_dtype=torch.float16)

HunyuanDiT2DControlNetModel

class diffusers.HunyuanDiT2DControlNetModel

< source >

( conditioning_channels: int = 3 num_attention_heads: int = 16 attention_head_dim: int = 88 in_channels: typing.Optional[int] = None patch_size: typing.Optional[int] = None activation_fn: str = 'gelu-approximate' sample_size = 32 hidden_size = 1152 transformer_num_layers: int = 40 mlp_ratio: float = 4.0 cross_attention_dim: int = 1024 cross_attention_dim_t5: int = 2048 pooled_projection_dim: int = 1024 text_len: int = 77 text_len_t5: int = 256 use_style_cond_and_image_meta_size: bool = True )

forward

< source >

( hidden_states timestep controlnet_cond: Tensor conditioning_scale: float = 1.0 encoder_hidden_states = None text_embedding_mask = None encoder_hidden_states_t5 = None text_embedding_mask_t5 = None image_meta_size = None style = None image_rotary_emb = None return_dict = True )

引數

hidden_states (形狀為 (batch size, dim, height, width) 的 torch.Tensor) — 輸入張量。
timestep (torch.LongTensor, 可選) — 用於指示去噪步驟。
controlnet_cond (torch.Tensor) — ControlNet 的條件輸入。
conditioning_scale (float) — 表示條件縮放。
encoder_hidden_states (形狀為 (batch size, sequence len, embed dims) 的 torch.Tensor, 可選) — 交叉注意力層的條件嵌入。這是 BertModel 的輸出。
text_embedding_mask — torch.Tensor 形狀為 (batch, key_tokens) 的注意力掩碼應用於 encoder_hidden_states。這是 BertModel 的輸出。
encoder_hidden_states_t5 (形狀為 (batch size, sequence len, embed dims) 的 torch.Tensor, 可選) — 交叉注意力層的條件嵌入。這是 T5 文字編碼器的輸出。
text_embedding_mask_t5 — torch.Tensor 形狀為 (batch, key_tokens) 的注意力掩碼應用於 encoder_hidden_states。這是 T5 文字編碼器的輸出。
image_meta_size (torch.Tensor) — 指示影像大小的條件嵌入
style — torch.Tensor：指示風格的條件嵌入
image_rotary_emb (torch.Tensor) — 在注意力計算過程中應用於查詢和鍵張量的影像旋轉嵌入。
return_dict — bool 是否返回字典。

HunyuanDiT2DControlNetModel 的 forward 方法。

設定注意力處理器

< source >

( processor: typing.Union[diffusers.models.attention_processor.AttnProcessor, diffusers.models.attention_processor.CustomDiffusionAttnProcessor, diffusers.models.attention_processor.AttnAddedKVProcessor, diffusers.models.attention_processor.AttnAddedKVProcessor2_0, diffusers.models.attention_processor.JointAttnProcessor2_0, diffusers.models.attention_processor.PAGJointAttnProcessor2_0, diffusers.models.attention_processor.PAGCFGJointAttnProcessor2_0, diffusers.models.attention_processor.FusedJointAttnProcessor2_0, diffusers.models.attention_processor.AllegroAttnProcessor2_0, diffusers.models.attention_processor.AuraFlowAttnProcessor2_0, diffusers.models.attention_processor.FusedAuraFlowAttnProcessor2_0, diffusers.models.attention_processor.FluxAttnProcessor2_0, diffusers.models.attention_processor.FluxAttnProcessor2_0_NPU, diffusers.models.attention_processor.FusedFluxAttnProcessor2_0, diffusers.models.attention_processor.FusedFluxAttnProcessor2_0_NPU, diffusers.models.attention_processor.CogVideoXAttnProcessor2_0, diffusers.models.attention_processor.FusedCogVideoXAttnProcessor2_0, diffusers.models.attention_processor.XFormersAttnAddedKVProcessor, diffusers.models.attention_processor.XFormersAttnProcessor, diffusers.models.attention_processor.XLAFlashAttnProcessor2_0, diffusers.models.attention_processor.AttnProcessorNPU, diffusers.models.attention_processor.AttnProcessor2_0, diffusers.models.attention_processor.MochiVaeAttnProcessor2_0, diffusers.models.attention_processor.MochiAttnProcessor2_0, diffusers.models.attention_processor.StableAudioAttnProcessor2_0, diffusers.models.attention_processor.HunyuanAttnProcessor2_0, diffusers.models.attention_processor.FusedHunyuanAttnProcessor2_0, diffusers.models.attention_processor.PAGHunyuanAttnProcessor2_0, diffusers.models.attention_processor.PAGCFGHunyuanAttnProcessor2_0, diffusers.models.attention_processor.LuminaAttnProcessor2_0, diffusers.models.attention_processor.FusedAttnProcessor2_0, diffusers.models.attention_processor.CustomDiffusionXFormersAttnProcessor, diffusers.models.attention_processor.CustomDiffusionAttnProcessor2_0, diffusers.models.attention_processor.SlicedAttnProcessor, diffusers.models.attention_processor.SlicedAttnAddedKVProcessor, diffusers.models.attention_processor.SanaLinearAttnProcessor2_0, diffusers.models.attention_processor.PAGCFGSanaLinearAttnProcessor2_0, diffusers.models.attention_processor.PAGIdentitySanaLinearAttnProcessor2_0, diffusers.models.attention_processor.SanaMultiscaleLinearAttention, diffusers.models.attention_processor.SanaMultiscaleAttnProcessor2_0, diffusers.models.attention_processor.SanaMultiscaleAttentionProjection, diffusers.models.attention_processor.IPAdapterAttnProcessor, diffusers.models.attention_processor.IPAdapterAttnProcessor2_0, diffusers.models.attention_processor.IPAdapterXFormersAttnProcessor, diffusers.models.attention_processor.SD3IPAdapterJointAttnProcessor2_0, diffusers.models.attention_processor.PAGIdentitySelfAttnProcessor2_0, diffusers.models.attention_processor.PAGCFGIdentitySelfAttnProcessor2_0, diffusers.models.attention_processor.LoRAAttnProcessor, diffusers.models.attention_processor.LoRAAttnProcessor2_0, diffusers.models.attention_processor.LoRAXFormersAttnProcessor, diffusers.models.attention_processor.LoRAAttnAddedKVProcessor, typing.Dict[str, typing.Union[diffusers.models.attention_processor.AttnProcessor, diffusers.models.attention_processor.CustomDiffusionAttnProcessor, diffusers.models.attention_processor.AttnAddedKVProcessor, diffusers.models.attention_processor.AttnAddedKVProcessor2_0, diffusers.models.attention_processor.JointAttnProcessor2_0, diffusers.models.attention_processor.PAGJointAttnProcessor2_0, diffusers.models.attention_processor.PAGCFGJointAttnProcessor2_0, diffusers.models.attention_processor.FusedJointAttnProcessor2_0, diffusers.models.attention_processor.AllegroAttnProcessor2_0, diffusers.models.attention_processor.AuraFlowAttnProcessor2_0, diffusers.models.attention_processor.FusedAuraFlowAttnProcessor2_0, diffusers.models.attention_processor.FluxAttnProcessor2_0, diffusers.models.attention_processor.FluxAttnProcessor2_0_NPU, diffusers.models.attention_processor.FusedFluxAttnProcessor2_0, diffusers.models.attention_processor.FusedFluxAttnProcessor2_0_NPU, diffusers.models.attention_processor.CogVideoXAttnProcessor2_0, diffusers.models.attention_processor.FusedCogVideoXAttnProcessor2_0, diffusers.models.attention_processor.XFormersAttnAddedKVProcessor, diffusers.models.attention_processor.XFormersAttnProcessor, diffusers.models.attention_processor.XLAFlashAttnProcessor2_0, diffusers.models.attention_processor.AttnProcessorNPU, diffusers.models.attention_processor.AttnProcessor2_0, diffusers.models.attention_processor.MochiVaeAttnProcessor2_0, diffusers.models.attention_processor.MochiAttnProcessor2_0, diffusers.models.attention_processor.StableAudioAttnProcessor2_0, diffusers.models.attention_processor.HunyuanAttnProcessor2_0, diffusers.models.attention_processor.FusedHunyuanAttnProcessor2_0, diffusers.models.attention_processor.PAGHunyuanAttnProcessor2_0, diffusers.models.attention_processor.PAGCFGHunyuanAttnProcessor2_0, diffusers.models.attention_processor.LuminaAttnProcessor2_0, diffusers.models.attention_processor.FusedAttnProcessor2_0, diffusers.models.attention_processor.CustomDiffusionXFormersAttnProcessor, diffusers.models.attention_processor.CustomDiffusionAttnProcessor2_0, diffusers.models.attention_processor.SlicedAttnProcessor, diffusers.models.attention_processor.SlicedAttnAddedKVProcessor, diffusers.models.attention_processor.SanaLinearAttnProcessor2_0, diffusers.models.attention_processor.PAGCFGSanaLinearAttnProcessor2_0, diffusers.models.attention_processor.PAGIdentitySanaLinearAttnProcessor2_0, diffusers.models.attention_processor.SanaMultiscaleLinearAttention, diffusers.models.attention_processor.SanaMultiscaleAttnProcessor2_0, diffusers.models.attention_processor.SanaMultiscaleAttentionProjection, diffusers.models.attention_processor.IPAdapterAttnProcessor, diffusers.models.attention_processor.IPAdapterAttnProcessor2_0, diffusers.models.attention_processor.IPAdapterXFormersAttnProcessor, diffusers.models.attention_processor.SD3IPAdapterJointAttnProcessor2_0, diffusers.models.attention_processor.PAGIdentitySelfAttnProcessor2_0, diffusers.models.attention_processor.PAGCFGIdentitySelfAttnProcessor2_0, diffusers.models.attention_processor.LoRAAttnProcessor, diffusers.models.attention_processor.LoRAAttnProcessor2_0, diffusers.models.attention_processor.LoRAXFormersAttnProcessor, diffusers.models.attention_processor.LoRAAttnAddedKVProcessor]]] )

引數

processor (AttentionProcessor 字典或僅 AttentionProcessor) — 例項化的處理器類或處理器類字典，將被設定為**所有** Attention 層的處理器。如果 processor 是字典，則鍵需要定義到相應交叉注意力處理器的路徑。強烈建議在設定可訓練注意力處理器時使用此方法。

設定用於計算注意力的注意力處理器。

< > 在 GitHub 上更新

←FluxControlNetModel SanaControlNetModel→