Diffusers 文件
CogVideoXTransformer3D模型
並獲得增強的文件體驗
開始使用
CogVideoXTransformer3D模型
清華大學和智譜AI在CogVideoX: Text-to-Video Diffusion Models with An Expert Transformer中介紹了來自CogVideoX的用於3D資料的擴散Transformer模型。
該模型可以透過以下程式碼片段載入。
from diffusers import CogVideoXTransformer3DModel
transformer = CogVideoXTransformer3DModel.from_pretrained("THUDM/CogVideoX-2b", subfolder="transformer", torch_dtype=torch.float16).to("cuda")CogVideoXTransformer3D模型
class diffusers.CogVideoXTransformer3DModel
< 源 >( num_attention_heads: int = 30 attention_head_dim: int = 64 in_channels: int = 16 out_channels: typing.Optional[int] = 16 flip_sin_to_cos: bool = True freq_shift: int = 0 time_embed_dim: int = 512 ofs_embed_dim: typing.Optional[int] = None text_embed_dim: int = 4096 num_layers: int = 30 dropout: float = 0.0 attention_bias: bool = True sample_width: int = 90 sample_height: int = 60 sample_frames: int = 49 patch_size: int = 2 patch_size_t: typing.Optional[int] = None temporal_compression_ratio: int = 4 max_text_seq_length: int = 226 activation_fn: str = 'gelu-approximate' timestep_activation_fn: str = 'silu' norm_elementwise_affine: bool = True norm_eps: float = 1e-05 spatial_interpolation_scale: float = 1.875 temporal_interpolation_scale: float = 1.0 use_rotary_positional_embeddings: bool = False use_learned_positional_embeddings: bool = False patch_bias: bool = True )
引數
- num_attention_heads (
int, defaults to30) — 用於多頭注意力機制的頭數。 - attention_head_dim (
int, defaults to64) — 每個注意力頭中的通道數。 - in_channels (
int, defaults to16) — 輸入中的通道數。 - out_channels (
int, 可選, defaults to16) — 輸出中的通道數。 - flip_sin_to_cos (
bool, defaults toTrue) — 是否在時間嵌入中將sin翻轉為cos。 - time_embed_dim (
int, defaults to512) — 時間步嵌入的輸出維度。 - ofs_embed_dim (
int, defaults to512) — CogVideoX-5b-I2B 1.5版中使用的“ofs”嵌入的輸出維度。 - text_embed_dim (
int, defaults to4096) — 文字編碼器中文字嵌入的輸入維度。 - num_layers (
int, defaults to30) — 要使用的Transformer塊層數。 - dropout (
float, defaults to0.0) — 要使用的dropout機率。 - attention_bias (
bool, defaults toTrue) — 是否在注意力投影層中使用偏置。 - sample_width (
int, defaults to90) — 輸入潛在的寬度。 - sample_height (
int, defaults to60) — 輸入潛在的高度。 - sample_frames (
int, defaults to49) — 輸入潛在的幀數。請注意，此引數最初錯誤地初始化為49而非13，因為CogVideoX在預設和推薦設定下一次性處理13個潛在幀，但為確保向後相容性，無法更改為正確的值。要建立具有K個潛在幀的Transformer，此處應傳遞的正確值為：((K - 1) * temporal_compression_ratio + 1)。 - patch_size (
int, defaults to2) — 補丁嵌入層中使用的補丁大小。 - temporal_compression_ratio (
int, defaults to4) — 跨時間維度的壓縮比。請參閱sample_frames的文件。 - max_text_seq_length (
int, defaults to226) — 輸入文字嵌入的最大序列長度。 - activation_fn (
str, defaults to"gelu-approximate") — 前饋網路中使用的啟用函式。 - timestep_activation_fn (
str, defaults to"silu") — 生成時間步嵌入時使用的啟用函式。 - norm_elementwise_affine (
bool, defaults toTrue) — 是否在歸一化層中使用逐元素仿射。 - norm_eps (
float, defaults to1e-5) — 歸一化層中使用的epsilon值。 - spatial_interpolation_scale (
float, defaults to1.875) — 在3D位置嵌入中應用於空間維度的縮放因子。 - temporal_interpolation_scale (
float, defaults to1.0) — 在3D位置嵌入中應用於時間維度的縮放因子。
在CogVideoX中用於影片類資料的Transformer模型。
設定注意力處理器
< 源 >( processor: typing.Union[diffusers.models.attention_processor.AttnProcessor, diffusers.models.attention_processor.CustomDiffusionAttnProcessor, diffusers.models.attention_processor.AttnAddedKVProcessor, diffusers.models.attention_processor.AttnAddedKVProcessor2_0, diffusers.models.attention_processor.JointAttnProcessor2_0, diffusers.models.attention_processor.PAGJointAttnProcessor2_0, diffusers.models.attention_processor.PAGCFGJointAttnProcessor2_0, diffusers.models.attention_processor.FusedJointAttnProcessor2_0, diffusers.models.attention_processor.AllegroAttnProcessor2_0, diffusers.models.attention_processor.AuraFlowAttnProcessor2_0, diffusers.models.attention_processor.FusedAuraFlowAttnProcessor2_0, diffusers.models.attention_processor.FluxAttnProcessor2_0, diffusers.models.attention_processor.FluxAttnProcessor2_0_NPU, diffusers.models.attention_processor.FusedFluxAttnProcessor2_0, diffusers.models.attention_processor.FusedFluxAttnProcessor2_0_NPU, diffusers.models.attention_processor.CogVideoXAttnProcessor2_0, diffusers.models.attention_processor.FusedCogVideoXAttnProcessor2_0, diffusers.models.attention_processor.XFormersAttnAddedKVProcessor, diffusers.models.attention_processor.XFormersAttnProcessor, diffusers.models.attention_processor.XLAFlashAttnProcessor2_0, diffusers.models.attention_processor.AttnProcessorNPU, diffusers.models.attention_processor.AttnProcessor2_0, diffusers.models.attention_processor.MochiVaeAttnProcessor2_0, diffusers.models.attention_processor.MochiAttnProcessor2_0, diffusers.models.attention_processor.StableAudioAttnProcessor2_0, diffusers.models.attention_processor.HunyuanAttnProcessor2_0, diffusers.models.attention_processor.FusedHunyuanAttnProcessor2_0, diffusers.models.attention_processor.PAGHunyuanAttnProcessor2_0, diffusers.models.attention_processor.PAGCFGHunyuanAttnProcessor2_0, diffusers.models.attention_processor.LuminaAttnProcessor2_0, diffusers.models.attention_processor.FusedAttnProcessor2_0, diffusers.models.attention_processor.CustomDiffusionXFormersAttnProcessor, diffusers.models.attention_processor.CustomDiffusionAttnProcessor2_0, diffusers.models.attention_processor.SlicedAttnProcessor, diffusers.models.attention_processor.SlicedAttnAddedKVProcessor, diffusers.models.attention_processor.SanaLinearAttnProcessor2_0, diffusers.models.attention_processor.PAGCFGSanaLinearAttnProcessor2_0, diffusers.models.attention_processor.PAGIdentitySanaLinearAttnProcessor2_0, diffusers.models.attention_processor.SanaMultiscaleLinearAttention, diffusers.models.attention_processor.SanaMultiscaleAttnProcessor2_0, diffusers.models.attention_processor.SanaMultiscaleAttentionProjection, diffusers.models.attention_processor.IPAdapterAttnProcessor, diffusers.models.attention_processor.IPAdapterAttnProcessor2_0, diffusers.models.attention_processor.IPAdapterXFormersAttnProcessor, diffusers.models.attention_processor.SD3IPAdapterJointAttnProcessor2_0, diffusers.models.attention_processor.PAGIdentitySelfAttnProcessor2_0, diffusers.models.attention_processor.PAGCFGIdentitySelfAttnProcessor2_0, diffusers.models.attention_processor.LoRAAttnProcessor, diffusers.models.attention_processor.LoRAAttnProcessor2_0, diffusers.models.attention_processor.LoRAXFormersAttnProcessor, diffusers.models.attention_processor.LoRAAttnAddedKVProcessor, typing.Dict[str, typing.Union[diffusers.models.attention_processor.AttnProcessor, diffusers.models.attention_processor.CustomDiffusionAttnProcessor, diffusers.models.attention_processor.AttnAddedKVProcessor, diffusers.models.attention_processor.AttnAddedKVProcessor2_0, diffusers.models.attention_processor.JointAttnProcessor2_0, diffusers.models.attention_processor.PAGJointAttnProcessor2_0, diffusers.models.attention_processor.PAGCFGJointAttnProcessor2_0, diffusers.models.attention_processor.FusedJointAttnProcessor2_0, diffusers.models.attention_processor.AllegroAttnProcessor2_0, diffusers.models.attention_processor.AuraFlowAttnProcessor2_0, diffusers.models.attention_processor.FusedAuraFlowAttnProcessor2_0, diffusers.models.attention_processor.FluxAttnProcessor2_0, diffusers.models.attention_processor.FluxAttnProcessor2_0_NPU, diffusers.models.attention_processor.FusedFluxAttnProcessor2_0, diffusers.models.attention_processor.FusedFluxAttnProcessor2_0_NPU, diffusers.models.attention_processor.CogVideoXAttnProcessor2_0, diffusers.models.attention_processor.FusedCogVideoXAttnProcessor2_0, diffusers.models.attention_processor.XFormersAttnAddedKVProcessor, diffusers.models.attention_processor.XFormersAttnProcessor, diffusers.models.attention_processor.XLAFlashAttnProcessor2_0, diffusers.models.attention_processor.AttnProcessorNPU, diffusers.models.attention_processor.AttnProcessor2_0, diffusers.models.attention_processor.MochiVaeAttnProcessor2_0, diffusers.models.attention_processor.MochiAttnProcessor2_0, diffusers.models.attention_processor.StableAudioAttnProcessor2_0, diffusers.models.attention_processor.HunyuanAttnProcessor2_0, diffusers.models.attention_processor.FusedHunyuanAttnProcessor2_0, diffusers.models.attention_processor.PAGHunyuanAttnProcessor2_0, diffusers.models.attention_processor.PAGCFGHunyuanAttnProcessor2_0, diffusers.models.attention_processor.LuminaAttnProcessor2_0, diffusers.models.attention_processor.FusedAttnProcessor2_0, diffusers.models.attention_processor.CustomDiffusionXFormersAttnProcessor, diffusers.models.attention_processor.CustomDiffusionAttnProcessor2_0, diffusers.models.attention_processor.SlicedAttnProcessor, diffusers.models.attention_processor.SlicedAttnAddedKVProcessor, diffusers.models.attention_processor.SanaLinearAttnProcessor2_0, diffusers.models.attention_processor.PAGCFGSanaLinearAttnProcessor2_0, diffusers.models.attention_processor.PAGIdentitySanaLinearAttnProcessor2_0, diffusers.models.attention_processor.SanaMultiscaleLinearAttention, diffusers.models.attention_processor.SanaMultiscaleAttnProcessor2_0, diffusers.models.attention_processor.SanaMultiscaleAttentionProjection, diffusers.models.attention_processor.IPAdapterAttnProcessor, diffusers.models.attention_processor.IPAdapterAttnProcessor2_0, diffusers.models.attention_processor.IPAdapterXFormersAttnProcessor, diffusers.models.attention_processor.SD3IPAdapterJointAttnProcessor2_0, diffusers.models.attention_processor.PAGIdentitySelfAttnProcessor2_0, diffusers.models.attention_processor.PAGCFGIdentitySelfAttnProcessor2_0, diffusers.models.attention_processor.LoRAAttnProcessor, diffusers.models.attention_processor.LoRAAttnProcessor2_0, diffusers.models.attention_processor.LoRAXFormersAttnProcessor, diffusers.models.attention_processor.LoRAAttnAddedKVProcessor]]] )
設定用於計算注意力的注意力處理器。
Transformer2DModelOutput
class diffusers.models.modeling_outputs.Transformer2DModelOutput
< 源 >( sample: torch.Tensor )
引數
- sample (
torch.Tensor, 形狀為(batch_size, num_channels, height, width)；如果Transformer2DModel是離散的，則為(batch size, num_vector_embeds - 1, num_latent_pixels)) — 在encoder_hidden_states輸入條件下輸出的隱藏狀態。如果是離散的，則返回未去噪的潛在畫素的機率分佈。
Transformer2DModel 的輸出。