ERNIE 4.5 VL 28B A3B

Exact model and requirements

Artifact:

Baidu / ERNIE-4.5-VL-28B-A3B-PT full native twelve-shard checkpoint · checkpoint baidu/ERNIE-4.5-VL-28B-A3B-PT

Publisher revision:

e3815e65c607ea211bfe21b46ab0cd264b76731c

Artifact digest:

Multi-file checkpoint has no publisher single-file whole-checkpoint digest. Mandatory recipe verifies all twelve shard SHA256s and every required auxiliary separately; no index digest is misrepresented as the weight digest.

Download size:

54.78 GiB

Memory requirement (unknown):

Unknown — No measured per-device/full-process peak for BF16 native ERNIE VL TP4 with one image and4K context.

System RAM / disk:

Unknown / Unknown

Placement:

supported sharding. Single-host TP4, native text20Q/4KV and vision16-head divisibility; heterogeneous64/64 expert implementation uses tensor-parallel fused experts. Encoder mode weights; not data-parallel replication. Per-GPU allocation/fit remains unknown. No CPU/disk model offload or expert parallelism requested. TP4 partitions supported tensors; other parameters and buffers may replicate.

KV cache:

auto: model BF16 attention K/V cache; no FP8 cache selected

Workload:

4,096 token context; Unknown; Unknown; concurrency 1; batch 1; cache unknown; repetitions Unknown.

Tokenizer:

Exact native Ernie4_5_VLTokenizer SentencePiece with native chat_template.json · e3815e65c607ea211bfe21b46ab0cd264b76731c

Sampling / reasoning:

Native generation_config supplies temperature 0.2, top_p 0.8, repetition_penalty 1 and zero frequency/presence penalties. Request passes those explicitly; top_k=-1 and min_p 0 are author choices. max_tokens 256. Exact native template receives enable_thinking=false, inserting an empty completed <think> block before generation. No separate reasoning parser is enabled. Any model-generated reasoning/content shares the 256-token cap; incomplete output fails this smoke check.

Runtime prerequisites:

Ubuntu 24.04 x86_64 host; Ubuntu 22.04 container; NVIDIA Linux driver >=575.57.08 for container CUDA 12.9.1; working NVIDIA Container Toolkit and Docker GPU access.

Official artifact · License

Baidu supplies the exact native model under Apache License 2.0. Preserve license, attribution and applicable copyright/NOTICE text; identify modifications and review patent/redistribution obligations. No legal clearance or upstream training-data rights equivalence is claimed.

• model-00001-of-00012.safetensors · 4.65 GiB · revision e3815e65c607ea211bfe21b46ab0cd264b76731c
• model-00002-of-00012.safetensors · 4.65 GiB · revision e3815e65c607ea211bfe21b46ab0cd264b76731c
• model-00003-of-00012.safetensors · 4.66 GiB · revision e3815e65c607ea211bfe21b46ab0cd264b76731c
• model-00004-of-00012.safetensors · 4.65 GiB · revision e3815e65c607ea211bfe21b46ab0cd264b76731c
• model-00005-of-00012.safetensors · 4.65 GiB · revision e3815e65c607ea211bfe21b46ab0cd264b76731c
• model-00006-of-00012.safetensors · 4.66 GiB · revision e3815e65c607ea211bfe21b46ab0cd264b76731c
• model-00007-of-00012.safetensors · 4.65 GiB · revision e3815e65c607ea211bfe21b46ab0cd264b76731c
• model-00008-of-00012.safetensors · 4.66 GiB · revision e3815e65c607ea211bfe21b46ab0cd264b76731c
• model-00009-of-00012.safetensors · 4.65 GiB · revision e3815e65c607ea211bfe21b46ab0cd264b76731c
• model-00010-of-00012.safetensors · 4.65 GiB · revision e3815e65c607ea211bfe21b46ab0cd264b76731c
• model-00011-of-00012.safetensors · 4.66 GiB · revision e3815e65c607ea211bfe21b46ab0cd264b76731c
• model-00012-of-00012.safetensors · 3.61 GiB · revision e3815e65c607ea211bfe21b46ab0cd264b76731c
• tokenizer.model · 0 GiB · revision e3815e65c607ea211bfe21b46ab0cd264b76731c
• config.json · 0 GiB · revision e3815e65c607ea211bfe21b46ab0cd264b76731c
• preprocessor_config.json · 0 GiB · revision e3815e65c607ea211bfe21b46ab0cd264b76731c
• chat_template.json · 0 GiB · revision e3815e65c607ea211bfe21b46ab0cd264b76731c

• All 58,817,660,680 original weight bytes load; 3B activated is a compute-routing property, not permission to load only 3B parameters.
• Four L40S devices each have vendor 48GB memory; exact byte quantity is unresolved. Aggregate memory is not a single-device allocation and weights/cache/vision/buffers may shard or replicate differently. Fit on each device remains unknown.
• TP4 satisfies documented head/intermediate/expert divisibility, not a measured startup, memory or performance result. GPU/NCCL topology and latency are unmeasured.

Install the runtime

1. 1. Prepare Ubuntu GPU access

   On this existing VM install Ubuntu-managed GPU drivers, then reboot and reconnect through your existing SSH access before the next step. Driver selection resolves from Ubuntu packages; the subsequent exact minimum/version check is mandatory. Do not install a second CUDA toolkit on the host.

   CommandCopy command

   sudo apt-get update
   sudo apt-get install -y ubuntu-drivers-common curl ca-certificates gnupg python3 openssl
   sudo ubuntu-drivers install --gpgpu
   sudo reboot

   Expected result: Host reboots. Reconnect using existing SSH; stop if NVIDIA driver installation fails.

2. 2. Install Docker and NVIDIA Container Toolkit

   Use official signed apt repositories. Host packages resolve at installation; record installed versions. These are reader-run system changes, not changes executed during documentation.

   CommandCopy command

   sudo install -m 0755 -d /etc/apt/keyrings
   sudo curl -fsSL https://download.docker.com/linux/ubuntu/gpg -o /etc/apt/keyrings/docker.asc
   sudo chmod a+r /etc/apt/keyrings/docker.asc
   . /etc/os-release
   printf 'deb [arch=amd64 signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/ubuntu %s stable
   ' "$VERSION_CODENAME" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
   sudo apt-get update
   sudo apt-get install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin
   curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg
   curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list | sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list > /dev/null
   sudo apt-get update
   sudo apt-get install -y nvidia-container-toolkit
   sudo nvidia-ctk runtime configure --runtime=docker
   sudo systemctl restart docker
   mkdir -p "$HOME/ernie-vl-hosting/model"
   cd "$HOME/ernie-vl-hosting"
   dpkg-query -W docker-ce nvidia-container-toolkit > host-packages.txt
   nvidia-smi --query-gpu=name,driver_version,memory.total --format=csv > gpu-inventory.txt
   cat gpu-inventory.txt

   Expected result: Docker and GPU runtime are installed; inventory lists exactly four L40S. Preserve package/driver/physical-memory evidence. Stop on driver/runtime errors.

3. 3. Build the pinned dependency layer and check GPU prerequisites

   The exact official linux/amd64 base uses CUDA 12.9.1. Native ERNIE processor unconditionally imports Decord, absent from the base dependency declarations, so build the documented minimal layer adding only its exact hash-pinned 0.6.0 wheel without dependency upgrades. Verify Dockerfile bytes, package imports, vLLM version, publisher Transformers upper bound, driver >=575.57.08 and four GPUs. This diagnostic loads no model. The derived image has no preclaimed binary digest; record its resulting image ID.

   CommandCopy command

   cd "$HOME/ernie-vl-hosting"
   python3 - <<'PYGPU'
   import subprocess
   rows=subprocess.check_output(['nvidia-smi','--query-gpu=name,driver_version','--format=csv,noheader'],text=True).splitlines()
   assert len(rows)==4
   for row in rows:
    name,version=[x.strip() for x in row.split(',')]
    assert 'L40S' in name
    assert tuple(map(int,version.split('.'))) >= (575,57,8)
   print('Four L40S and minimum driver verified')
   PYGPU
   sudo docker pull --platform linux/amd64 vllm/vllm-openai@sha256:47a9896f86818fea323b2d38082758c62d9a0155d6fe6c4dbd7d735c556f680a
   cat > Dockerfile <<'DOCKERFILE'
   FROM vllm/vllm-openai@sha256:47a9896f86818fea323b2d38082758c62d9a0155d6fe6c4dbd7d735c556f680a
   RUN python -m pip install --no-deps 'https://files.pythonhosted.org/packages/11/79/936af42edf90a7bd4e41a6cac89c913d4b47fa48a26b042d5129a9242ee3/decord-0.6.0-py3-none-manylinux2010_x86_64.whl#sha256=51997f20be8958e23b7c4061ba45d0efcd86bffd5fe81c695d0befee0d442976' && python -m pip check
   DOCKERFILE
   sha256sum --check <<'SHA256'
   f2bd752dd92f07393af2a3053af97369c5e05692e6ebd52e266c1cfecbdfbe8e  Dockerfile
   SHA256
   sudo docker build --pull=false --tag ernie-vl-image:local --file Dockerfile .
   sudo docker run --rm --gpus all --entrypoint python ernie-vl-image:local -c "import vllm,torch,transformers,decord,sentencepiece,requests,numpy,PIL; assert vllm.__version__=='0.11.2'; assert decord.__version__=='0.6.0'; assert tuple(map(int,transformers.__version__.split('.'))) <= (4,57,6); assert torch.cuda.device_count()==4; print(vllm.__version__,torch.__version__,torch.version.cuda,transformers.__version__); print(torch.cuda.get_device_properties(0))" > container-environment.txt
   cat container-environment.txt
   sudo docker image inspect ernie-vl-image:local --format '{{.Id}}' > derived-image-id.txt

   Expected result: Dockerfile checksum and pip check pass; derived image ID is recorded. vLLM 0.11.2, Decord 0.6.0, Transformers <=4.57.6 and four GPUs pass. Stop on import, driver, package or build failure. No model fit is tested.

Download the model

1. 1. Fetch and verify all twelve original weight shards

   This downloads the full native58,817,660,680-byte BF16 weights at one immutable revision. Existing valid files are reused; invalid partials never replace final files. Stop on any mismatch. No active-parameter memory discount or quantizer is assumed.

   CommandCopy command

   cd "$HOME/ernie-vl-hosting"
   python3 - <<'PYDOWNLOAD'
   import pathlib,json,hashlib,urllib.request
   manifest=json.loads("{\"model-00001-of-00012.safetensors\":{\"bytes\":4991358608,\"sha256\":\"9bf21cc35126cba976658743bf6cb834592d62b054df707949aefe54a8819fc1\"},\"model-00002-of-00012.safetensors\":{\"bytes\":4988732080,\"sha256\":\"3be2e55d382fd35af01b352ec954d70d9c85e2c9773318873160262eaca94521\"},\"model-00003-of-00012.safetensors\":{\"bytes\":4999222944,\"sha256\":\"f015a401233aa4aca5ea6d84fc852ceb8b013ce69906ef8e9363b852908a74ff\"},\"model-00004-of-00012.safetensors\":{\"bytes\":4995307456,\"sha256\":\"23c836804da9ffc4fce5ec8514a3e342f6c419ea0919e82722a0d29970cf8462\"},\"model-00005-of-00012.safetensors\":{\"bytes\":4988732984,\"sha256\":\"8d0851b85a92ea291f4735588ec5ebd7d9d09700d8f9805995563f1ef0c80e84\"},\"model-00006-of-00012.safetensors\":{\"bytes\":4999234272,\"sha256\":\"40f787a8377b43959b388d7cc87c8ee8f4edef03c11441a8d2e1db46d1e909f0\"},\"model-00007-of-00012.safetensors\":{\"bytes\":4995298008,\"sha256\":\"4747685c94cb90f9b52233bbf5bc54d037860d90d73771b26b94b53ede995bee\"},\"model-00008-of-00012.safetensors\":{\"bytes\":4999219488,\"sha256\":\"e3d5b703c1f51475fcfdbb1bdad942ed3799dbb3486cb392765d9d167a2378cc\"},\"model-00009-of-00012.safetensors\":{\"bytes\":4995312664,\"sha256\":\"9fedce1274b09ffb472c999e7954a1152fca9bcc11a3e1ceaee2811c5798a9da\"},\"model-00010-of-00012.safetensors\":{\"bytes\":4988732992,\"sha256\":\"6ec0f5133e2cf9170ae86fa8b4edf4c57edff9b69ecff1c9d60d4b6990fb8f2b\"},\"model-00011-of-00012.safetensors\":{\"bytes\":4999229776,\"sha256\":\"5b62486df379ff3bacca4a3387c6e2d77158d7c55b5711f17a52c384ad57b2aa\"},\"model-00012-of-00012.safetensors\":{\"bytes\":3877279408,\"sha256\":\"c67ec8e731d393ba6c945290fc758a7e132115c30c2328428af9bdfdf5070082\"}}")
   root=pathlib.Path('model');root.mkdir(exist_ok=True)
   for name,meta in manifest.items():
    path=root/name
    def valid(p):
     if not p.exists() or p.stat().st_size!=meta['bytes']: return False
     h=hashlib.sha256()
     with p.open('rb') as f:
      for b in iter(lambda:f.read(8*1024*1024),b''):h.update(b)
     return h.hexdigest()==meta['sha256']
    if not valid(path):
     tmp=path.with_name(path.name+'.partial')
     with urllib.request.urlopen('https://huggingface.co/baidu/ERNIE-4.5-VL-28B-A3B-PT/resolve/e3815e65c607ea211bfe21b46ab0cd264b76731c/'+name) as response,tmp.open('wb') as out:
      while True:
       b=response.read(8*1024*1024)
       if not b:break
       out.write(b)
     assert valid(tmp),name+' checksum/size mismatch'
     tmp.replace(path)
    assert valid(path),name
    print(name,'OK')
   PYDOWNLOAD

   Expected result: All twelve files report OK with exact byte lengths and LFS SHA256 digests. No model executes.

2. 2. Fetch and verify every required native auxiliary

   Download the exact index, config, tokenizer, special tokens, generation settings, template, remote configuration/model/processor source, README and LICENSE. The artifact auxiliary list is bounded to16 entries; this mandatory manifest pins the additional files fully rather than dropping their verification.

   CommandCopy command

   cd "$HOME/ernie-vl-hosting"
   python3 - <<'PYDOWNLOAD'
   import pathlib,json,hashlib,urllib.request
   manifest=json.loads("{\"LICENSE\":{\"bytes\":11365,\"sha256\":\"e06c1039021830c5531e2e5c50a7f29a34c0fd8f7f6c539620164d4e09ec89f9\"},\"README.md\":{\"bytes\":8000,\"sha256\":\"969e6de30e2d7ad36518adabbe0a78e1c69cc0b10e8fbeea0fa4ea0ddde5d3b3\"},\"added_tokens.json\":{\"bytes\":86904,\"sha256\":\"feea71fd871e27758410175c945fe72306fc0abedcfc4ddc5b47b988bd04497b\"},\"chat_template.json\":{\"bytes\":3187,\"sha256\":\"fde9aa8e9811392aa81cd7ee9c7b66fdf2345e4fe94977227427a1ecc532f73e\"},\"config.json\":{\"bytes\":1931,\"sha256\":\"16bc6e8c32957a78360dda7dde9b4b63b6641455a0d68090a5e19c9a5ef79a30\"},\"configuration_ernie4_5_vl.py\":{\"bytes\":28453,\"sha256\":\"2c6d9a10d958c61371f27f45bb4e005a03fd96af154ffda1ecf8c4d5e46ec27a\"},\"generation_config.json\":{\"bytes\":228,\"sha256\":\"82b75367512a69c9882a37b68a92b722e95386af59a432d929226ccd7840ed4f\"},\"model.safetensors.index.json\":{\"bytes\":995503,\"sha256\":\"928e6000ce3872249a214b746c5e4d843ee8ce7a783224cca90d24d156b7e3d4\"},\"modeling_ernie4_5_vl.py\":{\"bytes\":158369,\"sha256\":\"1d95250828add9a4752f04b9d0a705810f107e32a6cd42ca71e21164d078a08d\"},\"preprocessor_config.json\":{\"bytes\":743,\"sha256\":\"daed0971ca41bfe6bbb15be465fbee10b8b4900a54d8d0f9edbf68b6e689218c\"},\"processing_ernie4_5_vl.py\":{\"bytes\":70800,\"sha256\":\"09b66a0114243db31285b90eef7552457b3a5318d6db1d638785d8a1bc817e60\"},\"special_tokens_map.json\":{\"bytes\":62694,\"sha256\":\"b1b92788fd1e393051d620f5e4daa28463e99f40409459869f6d08ce4330f1d7\"},\"tokenizer.model\":{\"bytes\":1614362,\"sha256\":\"2ed2203974453df691287a0432c06737f1b17f20f5ab325fb33e31844d90ddb0\"},\"tokenizer_config.json\":{\"bytes\":3766,\"sha256\":\"f47d88241265adfb53508e7143ba3049cead2f41c8cd6bb81479a747e0c64220\"}}")
   root=pathlib.Path('model');root.mkdir(exist_ok=True)
   for name,meta in manifest.items():
    path=root/name
    def valid(p):
     if not p.exists() or p.stat().st_size!=meta['bytes']: return False
     h=hashlib.sha256()
     with p.open('rb') as f:
      for b in iter(lambda:f.read(8*1024*1024),b''):h.update(b)
     return h.hexdigest()==meta['sha256']
    if not valid(path):
     tmp=path.with_name(path.name+'.partial')
     with urllib.request.urlopen('https://huggingface.co/baidu/ERNIE-4.5-VL-28B-A3B-PT/resolve/e3815e65c607ea211bfe21b46ab0cd264b76731c/'+name) as response,tmp.open('wb') as out:
      while True:
       b=response.read(8*1024*1024)
       if not b:break
       out.write(b)
     assert valid(tmp),name+' checksum/size mismatch'
     tmp.replace(path)
    assert valid(path),name
    print(name,'OK')
   PYDOWNLOAD

   Expected result: Every remaining native file reports OK. No dependency or remote source is silently fetched from a different model revision.

Configure

1. 1. Prepare a private credential and extract the exact native template

   Preserve an existing nonempty service key. The environment file is private and Docker reads it without displaying a key in the command line. Privileged local users can inspect container environments; use a trusted VM. Extract the JSON string verbatim without stripping or adding a newline.

   CommandCopy command

   cd "$HOME/ernie-vl-hosting"
   umask 077
   if [ ! -e .api-key ]; then openssl rand -hex 32 > .api-key; fi
   test -s .api-key || { echo 'Restore empty service key before proceeding'; exit 1; }
   chmod 600 .api-key
   python3 - <<'PYCONFIG'
   import pathlib,json,hashlib
   key=pathlib.Path('.api-key').read_text().strip()
   assert len(key)==64 and all(c in '0123456789abcdef' for c in key)
   pathlib.Path('.service.env').write_text('VLLM_API_KEY='+key+'\n')
   raw=json.loads(pathlib.Path('model/chat_template.json').read_text())['chat_template'].encode()
   assert hashlib.sha256(raw).hexdigest()=='7194ad21096a9e62f04f1fc2d10770f89a82b970a7b062b27880434b1200ec39'
   pathlib.Path('native-chat.jinja').write_bytes(raw)
   print('Private key preserved; exact native template verified')
   PYCONFIG
   chmod 600 .service.env
   mkdir -p cache

   Expected result: Private service key/env file and exact native template exist. No secret is printed. The source template is not replaced with a generic chat format.

Start the model

1. 1. Start one private TP4 image service

   Use the verified reader-built image from the pinned official amd64 base and exact dependency layer and full local snapshot. Text attention, MoE expert tensors and vision attention have native TP4 paths; parameters/buffers can still replicate. Encoder mode weights and standard NCCL collectives are explicit. One sequence, 4K total context and bounded processor area are author workload choices. Request-level processor kwargs can override server defaults, so only trusted clients may call this private service.

   CommandCopy command

   cd "$HOME/ernie-vl-hosting"
   sudo docker run --rm --name ernie-vl-image --platform linux/amd64 --gpus all --shm-size=16g -p 127.0.0.1:8000:8000 --env-file .service.env -e HF_HUB_OFFLINE=1 -e TRANSFORMERS_OFFLINE=1 -v "$PWD/model:/model:ro" -v "$PWD/native-chat.jinja:/native-chat.jinja:ro" -v "$PWD/cache:/root/.cache" ernie-vl-image:local /model --served-model-name ernie-vl --host 0.0.0.0 --port 8000 --trust-remote-code --dtype bfloat16 --kv-cache-dtype auto --tensor-parallel-size 4 --distributed-executor-backend mp --disable-custom-all-reduce --mm-encoder-tp-mode weights --max-model-len 4096 --max-num-seqs 1 --gpu-memory-utilization 0.85 --enforce-eager --generation-config vllm --chat-template /native-chat.jinja --limit-mm-per-prompt '{"image":1,"video":0}' --mm-processor-kwargs '{"min_pixels":3136,"max_pixels":200704,"image_min_pixels":3136,"image_max_pixels":200704}'

   Expected result: Foreground server initializes all four workers and private host mapping. Stop on OOM, unsupported model/processor/backend, missing imports or TP/collective errors. Startup/fit is a reader check, not an observed result.

Verify the setup

1. 1. Verify readiness and the pinned real image request

   Run in a second SSH terminal on this VM. The inspected fixture is a newspaper front page with the large headline MEN WALK ON MOON and a lunar photograph. It comes from the pinned official llama.cpp test repository; repository inclusion does not establish rights to redistribute the newspaper image. Fetch only the source URL for this private check; do not rehost it. At the chosen200704-pixel cap, visual grid area is bounded to at most 256 nominal tokens (patch14, spatial merge2), before control/text overhead. Actual full token usage is unknown until returned.

   CommandCopy command

   cd "$HOME/ernie-vl-hosting"
   curl --fail http://127.0.0.1:8000/health
   curl --fail --location --output fixture.jpeg 'https://raw.githubusercontent.com/ggml-org/llama.cpp/620d2225cc436ab0b001e3200e590e3b286ef9ab/tools/mtmd/test-1.jpeg'
   sha256sum --check <<'SHA256'
   2dff664c0c8aaea18aff8cbe7e868845b775e90cdd7a0bac98df709b131deaa3  fixture.jpeg
   SHA256
   python3 - <<'PYREQUEST'
   import pathlib,json,base64,hashlib
   raw=pathlib.Path('fixture.jpeg').read_bytes()
   assert len(raw)==124071 and hashlib.sha256(raw).hexdigest()=='2dff664c0c8aaea18aff8cbe7e868845b775e90cdd7a0bac98df709b131deaa3'
   request=json.loads("{\"model\":\"ernie-vl\",\"messages\":[{\"role\":\"user\",\"content\":[{\"type\":\"text\",\"text\":\"What is the largest headline near the top of this image, and what kind of document is shown? Give a short answer based on the image.\"},{\"type\":\"image_url\",\"image_url\":{\"url\":\"REPLACE_DATA_URI\"}}]}],\"max_tokens\":256,\"temperature\":0.2,\"top_p\":0.8,\"top_k\":-1,\"min_p\":0,\"repetition_penalty\":1.0,\"frequency_penalty\":0.0,\"presence_penalty\":0.0,\"chat_template_kwargs\":{\"enable_thinking\":false},\"mm_processor_kwargs\":{\"min_pixels\":3136,\"max_pixels\":200704,\"image_min_pixels\":3136,\"image_max_pixels\":200704},\"stop\":[\"<|end_of_sentence|>\"]}")
   request['messages'][0]['content'][1]['image_url']['url']='data:image/jpeg;base64,'+base64.b64encode(raw).decode('ascii')
   pathlib.Path('image-request.json').write_text(json.dumps(request))
   PYREQUEST
   ERNIE_VL_KEY="$(cat .api-key)"
   curl --fail http://127.0.0.1:8000/v1/chat/completions -H "Authorization: Bearer $ERNIE_VL_KEY" -H 'Content-Type: application/json' --data-binary @image-request.json --output image-response.json
   cat image-response.json
   python3 - <<'PYRESULT'
   import json
   r=json.load(open('image-response.json'))
   c=r['choices'][0]
   assert isinstance(c['message']['content'],str) and c['message']['content'].strip()
   assert c['finish_reason']!='length'
   u=r['usage'];assert u['prompt_tokens']+u['completion_tokens']<=4096 and 1<=u['completion_tokens']<=256
   print('Inspect answer for newspaper and MEN WALK ON MOON; exact prose is unpromised')
   PYRESULT
   curl -s -o /dev/null -w '%{http_code}\n' http://127.0.0.1:8000/v1/chat/completions -H 'Content-Type: application/json' --data-binary @image-request.json
   unset ERNIE_VL_KEY

   Expected result: Ready health returns200, both fixture checks pass, authenticated image generation gives a complete short answer identifying a newspaper and the headline MEN WALK ON MOON, and no-key chat returns401. Exact prose and measured quality/speed are not promised. Wrong/unreadable headline, unfinished reasoning, missing content, length finish or context/worker failure fails the smoke check.

Performance evidence

No compatible measurement available. Documented support and estimated memory do not establish response speed.

Limits and troubleshooting

• Source-documented only. No weights, image layers, installation, build, model import/inference, fit, speed or quality measurement was performed.
• Exact native vLLM architecture registration is independent of the unsupported llama.cpp ERNIE-VL path and the source-level native Python CPU cache/mask concern. This GPU recipe uses the dedicated vLLM implementation, not a patch to native remote modeling code.
• Native28B total/3B active heterogeneous text+vision MoE loads all original weights. Vision/resampler are integrated; no separate mmproj binary is needed or invented.
• TP4 is source-supported by text 20Q/4KV heads, vision 16 heads and heterogeneous 64/64-expert tensor paths with intermediate 1536/512 dimensions. Divisibility does not establish per-device memory fit or performance.
• AWS gives48GB per L40S without exact byte units in the captured source. Per-device memory bytes are unknown, not48GiB and not an aggregate 192GiB pool. Some tensors/buffers replicate.
• Container tag0.11.2 is resolved to an immutable amd64 manifest. Its metadata uses CUDA 12.9.1; require driver >=575.57.08. General vLLM docs mention12.8, so those generic defaults are not silently substituted. Actual installed Torch/Transformers/CUDA versions are printed by the reader diagnostic and remain unexecuted here.
• The reader-built image adds only the exact Decord 0.6.0 manylinux x86_64 wheel by SHA256 to the immutable official base; its Dockerfile is hashed and resulting image ID is recorded. No build was executed by the author. Imports, pip check and publisher Transformers <=4.57.6 bound must pass before model loading; unpinned fixes require a new runtime qualification.
• Both image processor min/max_pixels and ERNIE processor image_min/image_max_pixels are explicitly bounded 3136..200704. vLLM lets request processor kwargs override server defaults; this guide passes the same bounds but does not claim immutable per-client security limits.
• One image is permitted and video count is zero for this job. Pixel area/(14*2)^2 gives at most 256 nominal visual grid tokens for a single image under the source formula; actual full input including markers/text, generated token count and total usage remain unmeasured. Native model supports 131072 positions but this recipe only allocates 4096.
• Native template is extracted verbatim from verified chat_template.json. Request enable_thinking=false inserts empty completed think tags; generated content remains capped256. No separate reasoning parser/tool executor or exact generated wording is claimed.
• The private fixture is manually inspected newspaper imagery from a pinned runtime source URL; underlying newspaper-image redistribution rights remain unknown. Do not assume the code repository license grants those media rights or rehost the fixture.
• 384 GiB host RAM and 300 GiB root EBS are allocations, not measured minima. Existing local NVMe is not used. GPU-memory-utilization 0.85 and enforce-eager are author workload choices, not fit guarantees.
• Original dated compute and gp3 quotes are preserved; actual availability, quota, installed system package versions, storage usage, egress, tax and administration are unknown. A stopped container does not stop VM/EBS billing.
• Only one cloud image job is qualified here. Native video has actual runtime support but needs a separate temporal fixture/frame budget; CPU/local-GPU, thinking, tools, full context and other offers remain open.

Driver/container GPU test fails

Stop before model loading. Verify four L40S devices, Linux driver >=575.57.08, NVIDIA Container Toolkit configuration and exact amd64 container digest. Preserve diagnostics; do not disable NVIDIA compatibility checks.

Missing processor import, model mapping or hash mismatch

Stop. Recheck the full immutable snapshot manifest, exact base digest, hashed Dockerfile and Decord wheel. No ERNIE text checkpoint, PaddleOCR model or Qwen projector is compatible by assumption. Do not install ad-hoc fixes beyond the documented dependency layer.

OOM, NCCL or TP startup failure

Stop and inspect per-device inventory, logs and host disk/RAM. Four vendor 48GB devices are not a single pool. A changed precision, device count, offload or runtime is a separate configuration requiring qualification.

Wrong/empty headline, unfinished reasoning or length finish

Treat the smoke check as failed. Inspect actual image processing, token usage and response. No measured headline accuracy or completed answer was promised; do not fabricate a successful visual result.

HTTP401 or connection refused

Check foreground container readiness, matching private key, host loopback mapping and existing SSH tunnel. Missing-key401 is expected; do not open public 8000 ingress.

Stop the model

1. 1. Stop service and tunnel

   Press Ctrl+C in the foreground Docker terminal; alternatively use the command below from another SSH terminal. Close any SSH tunnel with Ctrl+C and unset client key variables. Stopping only this container does not stop EC2 billing.

   CommandCopy command

   sudo docker stop ernie-vl-image
   unset ERNIE_VL_KEY

   Expected result: Container exits and host loopback 8000 no longer serves. --rm removes the exited container, not downloaded images/model files.

Uninstall

1. 1. Remove dedicated files and optional container image

   Stop the container first, preserve wanted diagnostic logs, verify the path and remove only this walkthrough directory. Docker/driver/toolkit are shared system dependencies; leave them for other workloads. Remove the pinned image only if no other workload uses it.

   CommandCopy command

   rm -r -- "$HOME/ernie-vl-hosting"
   sudo docker image rm ernie-vl-image:local
   sudo docker image rm vllm/vllm-openai@sha256:47a9896f86818fea323b2d38082758c62d9a0155d6fe6c4dbd7d735c556f680a

   Expected result: Dedicated weights, cache, template, fixture/request/response, logs and private credential files are removed. Shared runtime packages remain.

Shut down cloud resources

1. 1. End selected cloud billing resources

   In EC2 us-east-1 stop this exact VM for reuse or terminate it. Review root EBS Delete on termination. Remove unwanted retained/detached EBS volumes, snapshots and public/Elastic IP resources; stopping VM compute leaves storage/retained resources billable. Check final Bills/Cost Explorer for actual usage.

   Expected result: Exact instance lifecycle transition completes; unwanted retained billed resources are removed. Stopping Docker alone does not stop VM compute charges.