InstructPix2Pix

概述

InstructPix2Pix：學習遵循影像編輯指令由 Tim Brooks、Aleksander Holynski 和 Alexei A. Efros 撰寫。

🤗 Optimum 擴充套件了 Diffusers 以支援在第二代 Neuron 裝置（支援 Trainium 和 Inferentia 2）上進行推理。它旨在繼承 Diffusers 在 Neuron 上的易用性。

匯出到 Neuron

要部署模型，您需要將它們編譯為針對 AWS Neuron 最佳化的 TorchScript。對於 Stable Diffusion，有四個元件需要匯出為 .neuron 格式以提高效能

文字編碼器
U-Net
VAE 編碼器
VAE 解碼器

您可以透過 CLI 或 NeuronStableDiffusionInstructPix2PixPipeline 類編譯和匯出 Stable Diffusion Checkpoint。

使用示例

使用 NeuronStableDiffusionInstructPix2PixPipeline 類，您可以利用文字和影像指導應用基於指令的影像編輯。

import requests
import PIL
from io import BytesIO
from optimum.neuron import NeuronStableDiffusionInstructPix2PixPipeline

def download_image(url):
    response = requests.get(url)
    return PIL.Image.open(BytesIO(response.content)).convert("RGB")

model_id = "timbrooks/instruct-pix2pix"
input_shapes = {"batch_size": 1, "height": 512, "width": 512}

pipe = NeuronStableDiffusionInstructPix2PixPipeline.from_pretrained(
  model_id, export=True, dynamic_batch_size=True, **input_shapes,
)
pipe.save_pretrained("sd_ip2p/")

img_url = "https://huggingface.co/datasets/diffusers/diffusers-images-docs/resolve/main/mountain.png"
init_image = download_image(img_url).resize((512, 512))

prompt = "Add a beautiful sunset"
image = pipe(prompt=prompt, image=init_image).images[0]
image.save("sunset_mountain.png")

`影像`	`提示`	輸出
	*新增美麗的日落*

NeuronStableDiffusionInstructPix2PixPipeline

class optimum.neuron.NeuronStableDiffusionInstructPix2PixPipeline

< 原始檔 >

( config: dict[str, typing.Any] configs: dict[str, 'PretrainedConfig'] neuron_configs: dict[str, 'NeuronDefaultConfig'] data_parallel_mode: typing.Literal['none', 'unet', 'transformer', 'all'] scheduler: diffusers.schedulers.scheduling_utils.SchedulerMixin | None vae_decoder: torch.jit._script.ScriptModule | NeuronModelVaeDecoder text_encoder: torch.jit._script.ScriptModule | NeuronModelTextEncoder | None = None text_encoder_2: torch.jit._script.ScriptModule | NeuronModelTextEncoder | None = None unet: torch.jit._script.ScriptModule | NeuronModelUnet | None = None transformer: torch.jit._script.ScriptModule | NeuronModelTransformer | None = None vae_encoder: torch.jit._script.ScriptModule | NeuronModelVaeEncoder | None = None image_encoder: torch.jit._script.ScriptModule | None = None safety_checker: torch.jit._script.ScriptModule | None = None tokenizer: transformers.models.clip.tokenization_clip.CLIPTokenizer | transformers.models.t5.tokenization_t5.T5Tokenizer | None = None tokenizer_2: transformers.models.clip.tokenization_clip.CLIPTokenizer | None = None feature_extractor: transformers.models.clip.feature_extraction_clip.CLIPFeatureExtractor | None = None controlnet: torch.jit._script.ScriptModule | list[torch.jit._script.ScriptModule]| NeuronControlNetModel | NeuronMultiControlNetModel | None = None requires_aesthetics_score: bool = False force_zeros_for_empty_prompt: bool = True add_watermarker: bool | None = None model_save_dir: str | pathlib.Path | tempfile.TemporaryDirectory | None = None model_and_config_save_paths: dict[str, tuple[str, pathlib.Path]] | None = None )

call

< 原始檔 >

( *args **kwargs )

您希望我們在 🤗Optimum-neuron 中支援其他擴散功能嗎？請在 Optimum-neuron Github 倉庫中提交問題或在 HuggingFace 社群論壇上與我們討論，謝謝 🤗！

AWS Trainium & Inferentia