在 TGI 中進行視覺語言模型推論

視覺語言模型（VLM）是同時接收圖像和文字輸入以生成文字的模型。

VLM 透過圖像與文字數據的組合進行訓練，能夠處理多種任務，例如圖像描述生成、視覺問答和視覺對話。

VLM 與其他純文字或純圖像模型不同之處在於，它們能夠處理長上下文，並且即便在多輪對話，甚至在某些情況下涉及多張圖像時，仍能生成連貫且與圖像相關的文字。

以下是視覺語言模型的幾個常見應用場景：

圖像描述生成 (Image Captioning)：給定一張圖像，生成描述該圖像的文字說明。
視覺問答 (VQA)：給定一張圖像以及關於該圖像的問題，生成問題的答案。
多模態對話 (Multimodal Dialog)：針對多輪的圖像與對話內容生成回應。
圖像資訊檢索 (Image Information Retrieval)：給定一張圖像，從中擷取相關資訊。

如何使用視覺語言模型？

Hugging Face Hub Python 函式庫

若要透過 Python 使用視覺語言模型進行推論，您可以使用 huggingface_hub 函式庫。InferenceClient 類別提供了一種與 Inference API 互動的簡易方式。圖像可以作為 URL 或 base64 編碼的字串傳遞。InferenceClient 將會自動偵測圖像格式。

from huggingface_hub import InferenceClient

client = InferenceClient(base_url="http://127.0.0.1:3000")
image = "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/rabbit.png"
prompt = f"![]({image})What is this a picture of?\n\n"
for token in client.text_generation(prompt, max_new_tokens=16, stream=True):
    print(token)

# This is a picture of an anthropomorphic rabbit in a space suit.

from huggingface_hub import InferenceClient
import base64
import requests
import io

client = InferenceClient(base_url="http://127.0.0.1:3000")

# read image from local file
image_path = "rabbit.png"
with open(image_path, "rb") as f:
    image = base64.b64encode(f.read()).decode("utf-8")

image = f"data:image/png;base64,{image}"
prompt = f"![]({image})What is this a picture of?\n\n"

for token in client.text_generation(prompt, max_new_tokens=10, stream=True):
    print(token)

# This is a picture of an anthropomorphic rabbit in a space suit.

或是透過 chat_completion 端點

from huggingface_hub import InferenceClient

client = InferenceClient(base_url="http://127.0.0.1:3000")

chat = client.chat_completion(
    messages=[
        {
            "role": "user",
            "content": [
                {"type": "text", "text": "Whats in this image?"},
                {
                    "type": "image_url",
                    "image_url": {
                        "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/rabbit.png"
                    },
                },
            ],
        },
    ],
    seed=42,
    max_tokens=100,
)

print(chat)
# ChatCompletionOutput(choices=[ChatCompletionOutputComplete(finish_reason='length', index=0, message=ChatCompletionOutputMessage(role='assistant', content=" The image you've provided features an anthropomorphic rabbit in spacesuit attire. This rabbit is depicted with human-like posture and movement, standing on a rocky terrain with a vast, reddish-brown landscape in the background. The spacesuit is detailed with mission patches, circuitry, and a helmet that covers the rabbit's face and ear, with an illuminated red light on the chest area.\n\nThe artwork style is that of a", name=None, tool_calls=None), logprobs=None)], created=1714589614, id='', model='llava-hf/llava-v1.6-mistral-7b-hf', object='text_completion', system_fingerprint='2.0.2-native', usage=ChatCompletionOutputUsage(completion_tokens=100, prompt_tokens=2943, total_tokens=3043))

或是使用 OpenAI 的客戶端函式庫

from openai import OpenAI

# init the client but point it to TGI
client = OpenAI(base_url="https://:3000/v1", api_key="-")

chat_completion = client.chat.completions.create(
    model="tgi",
    messages=[
        {
            "role": "user",
            "content": [
                {"type": "text", "text": "Whats in this image?"},
                {
                    "type": "image_url",
                    "image_url": {
                        "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/rabbit.png"
                    },
                },
            ],
        },
    ],
    stream=False,
)

print(chat_completion)
# ChatCompletion(id='', choices=[Choice(finish_reason='eos_token', index=0, logprobs=None, message=ChatCompletionMessage(content=' The image depicts an anthropomorphic rabbit dressed in a space suit with gear that resembles NASA attire. The setting appears to be a solar eclipse with dramatic mountain peaks and a partial celestial body in the sky. The artwork is detailed and vivid, with a warm color palette and a sense of an adventurous bunny exploring or preparing for a journey beyond Earth. ', role='assistant', function_call=None, tool_calls=None))], created=1714589732, model='llava-hf/llava-v1.6-mistral-7b-hf', object='text_completion', system_fingerprint='2.0.2-native', usage=CompletionUsage(completion_tokens=84, prompt_tokens=2943, total_tokens=3027))

透過發送 cURL 請求進行推論

若要在 curl 中使用 generate_stream 端點，您可以加上 -N 旗標。此旗標會停用 curl 的預設緩衝功能，並在資料從伺服器送達時即時顯示。

curl -N 127.0.0.1:3000/generate_stream \
    -X POST \
    -d '{"inputs":"![](https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/rabbit.png)What is this a picture of?\n\n","parameters":{"max_new_tokens":16, "seed": 42}}' \
    -H 'Content-Type: application/json'

# ...
# data:{"index":16,"token":{"id":28723,"text":".","logprob":-0.6196289,"special":false},"generated_text":"This is a picture of an anthropomorphic rabbit in a space suit.","details":null}

透過 JavaScript 進行推論

首先，我們需要安裝 @huggingface/inference 函式庫。

npm install @huggingface/inference

無論您是使用 Inference Providers（我們的無伺服器 API）還是 Inference Endpoints，您都可以呼叫 InferenceClient。

我們可以透過提供端點 URL 和 Hugging Face 存取權杖來建立 InferenceClient。

import { InferenceClient } from "@huggingface/inference";

const client = new InferenceClient('hf_YOUR_TOKEN', { endpointUrl: 'https://YOUR_ENDPOINT.endpoints.huggingface.cloud' });

const prompt =
  "![](https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/rabbit.png)What is this a picture of?\n\n";

const stream = client.textGenerationStream({
  inputs: prompt,
  parameters: { max_new_tokens: 16, seed: 42 },
});
for await (const r of stream) {
  // yield the generated token
  process.stdout.write(r.token.text);
}

// This is a picture of an anthropomorphic rabbit in a space suit.

結合視覺語言模型與其他功能

TGI 中的 VLM 具有多項優勢，例如這些模型可以與其他功能搭配使用，以處理更複雜的任務。例如，您可以將 VLM 與引導式生成 (Guided Generation) 結合，從圖像中生成特定的 JSON 資料。

例如，我們可以從兔子圖像中擷取資訊，並生成一個包含位置、活動、觀察到的動物數量以及觀察到的動物種類的 JSON 物件。其結果看起來會像這樣：

{
  "activity": "Standing",
  "animals": ["Rabbit"],
  "animals_seen": 1,
  "location": "Rocky surface with mountains in the background and a red light on the rabbit's chest"
}

我們只需要將 JSON 結構提供給 VLM 模型，它就會為我們生成該 JSON 物件。

curl localhost:3000/generate \
    -X POST \
    -H 'Content-Type: application/json' \
    -d '{
    "inputs":"![](https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/rabbit.png)What is this a picture of?\n\n",
    "parameters": {
        "max_new_tokens": 100,
        "seed": 42,
        "grammar": {
            "type": "json",
            "value": {
                "properties": {
                    "location": {
                        "type": "string"
                    },
                    "activity": {
                        "type": "string"
                    },
                    "animals_seen": {
                        "type": "integer",
                        "minimum": 1,
                        "maximum": 5
                    },
                    "animals": {
                        "type": "array",
                        "items": {
                            "type": "string"
                        }
                    }
                },
                "required": ["location", "activity", "animals_seen", "animals"]
            }
        }
    }
}'

# {
#   "generated_text": "{ \"activity\": \"Standing\", \"animals\": [ \"Rabbit\" ], \"animals_seen\": 1, \"location\": \"Rocky surface with mountains in the background and a red light on the rabbit's chest\" }"
# }

想進一步了解視覺語言模型是如何運作的嗎？請參考這篇關於此主題的精彩部落格文章。