vllm0.24.0-tsingmicro-tsm260610

Prerequisites link

• Architecture: x86_64
• Chip models: Tsingmicro TX8110
• Host driver: 260610164501.01
• Container toolkit (optional) ⓘ: tx-container-toolkit >= 2.5.0

Image contents link

Built on link

Python link

3.10

Application package link

vllm==0.24.0+flagos

vllm-plugin-fl==0.3.0

Launch link

Published: harbor.baai.ac.cn/flagos-app/vllm0.24.0-tsingmicro-tsm260610:2.2.0-0.3.0

The image name is long — assign it to a variable first:

  IMG=harbor.baai.ac.cn/flagos-app/vllm0.24.0-tsingmicro-tsm260610:2.2.0-0.3.0
  

The two approaches below are alternatives — pick the one that matches how your host runs containers:

With the container toolkit link

Start an interactive shell:

  docker run --rm -it \
  --runtime=tsingmicro \
  -e TSINGMICRO_VISIBLE_DEVICES=all \
  $IMG bash
  

Start the app with its default settings:

  docker run --rm -it \
  --runtime=tsingmicro \
  -e TSINGMICRO_VISIBLE_DEVICES=all \
  $IMG
  

Pass arguments to the launcher:

  docker run --rm -it \
  --runtime=tsingmicro \
  -e TSINGMICRO_VISIBLE_DEVICES=all \
  $IMG vllm-serve --model <path> --port 9000
  

Without a toolkit — plain docker / podman link

Start an interactive shell:

  docker run --rm -it \
  --device /dev/accel \
  --device /dev/accel_drv_mgr \
  $IMG bash
  

Start the app with its default settings:

  docker run --rm -it \
  --device /dev/accel \
  --device /dev/accel_drv_mgr \
  $IMG
  

Pass arguments to the launcher:

  docker run --rm -it \
  --device /dev/accel \
  --device /dev/accel_drv_mgr \
  $IMG vllm-serve --model <path> --port 9000