使用量化模型 (dtypes)

在 Transformers.js v3 之前，我們使用 quantized 選項來指定是使用模型的量化 (q8) 版本還是全精度 (fp32) 版本，透過將 quantized 分別設定為 true 或 false 來實現。現在，我們新增了使用 dtype 引數從更長的列表中進行選擇的功能。

可用的量化型別列表取決於具體模型，但一些常見的型別包括：全精度 ("fp32")、半精度 ("fp16")、8 位 ("q8", "int8", "uint8") 和 4 位 ("q4", "bnb4", "q4f16")。

Available dtypes for mixedbread-ai/mxbai-embed-xsmall-v1 （例如，mixedbread-ai/mxbai-embed-xsmall-v1）

基本用法

示例： 以 4 位量化執行 Qwen2.5-0.5B-Instruct (演示)

import { pipeline } from "@huggingface/transformers";

// Create a text generation pipeline
const generator = await pipeline(
  "text-generation",
  "onnx-community/Qwen2.5-0.5B-Instruct",
  { dtype: "q4", device: "webgpu" },
);

// Define the list of messages
const messages = [
  { role: "system", content: "You are a helpful assistant." },
  { role: "user", content: "Tell me a funny joke." },
];

// Generate a response
const output = await generator(messages, { max_new_tokens: 128 });
console.log(output[0].generated_text.at(-1).content);

按模組設定 dtypes

一些編碼器-解碼器模型，如 Whisper 或 Florence-2，對量化設定（尤其是編碼器的設定）極為敏感。因此，我們添加了按模組選擇 dtypes 的功能，可以透過提供一個從模組名稱到 dtype 的對映來實現。

示例： 在 WebGPU 上執行 Florence-2 (演示)

import { Florence2ForConditionalGeneration } from "@huggingface/transformers";

const model = await Florence2ForConditionalGeneration.from_pretrained(
  "onnx-community/Florence-2-base-ft",
  {
    dtype: {
      embed_tokens: "fp16",
      vision_encoder: "fp16",
      encoder_model: "q4",
      decoder_model_merged: "q4",
    },
    device: "webgpu",
  },
);

Florence-2 running on WebGPU

檢視完整程式碼示例

import {
  Florence2ForConditionalGeneration,
  AutoProcessor,
  AutoTokenizer,
  RawImage,
} from "@huggingface/transformers";

// Load model, processor, and tokenizer
const model_id = "onnx-community/Florence-2-base-ft";
const model = await Florence2ForConditionalGeneration.from_pretrained(
  model_id,
  {
    dtype: {
      embed_tokens: "fp16",
      vision_encoder: "fp16",
      encoder_model: "q4",
      decoder_model_merged: "q4",
    },
    device: "webgpu",
  },
);
const processor = await AutoProcessor.from_pretrained(model_id);
const tokenizer = await AutoTokenizer.from_pretrained(model_id);

// Load image and prepare vision inputs
const url = "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/car.jpg";
const image = await RawImage.fromURL(url);
const vision_inputs = await processor(image);

// Specify task and prepare text inputs
const task = "<MORE_DETAILED_CAPTION>";
const prompts = processor.construct_prompts(task);
const text_inputs = tokenizer(prompts);

// Generate text
const generated_ids = await model.generate({
  ...text_inputs,
  ...vision_inputs,
  max_new_tokens: 100,
});

// Decode generated text
const generated_text = tokenizer.batch_decode(generated_ids, {
  skip_special_tokens: false,
})[0];

// Post-process the generated text
const result = processor.post_process_generation(
  generated_text,
  task,
  image.size,
);
console.log(result);
// { '<MORE_DETAILED_CAPTION>': 'A green car is parked in front of a tan building. The building has a brown door and two brown windows. The car is a two door and the door is closed. The green car has black tires.' }

< > 在 GitHub 上更新