Qwen3.5-9B

Exact model and requirements

Artifact:

Unsloth (GGUF quantization); Qwen (base model) / Qwen3.5-9B Q4_K_M, Unsloth derivative · checkpoint unsloth/Qwen3.5-9B-GGUF

Publisher revision:

3885219b6810b007914f3a7950a8d1b469d598a5

Artifact digest:

sha256:03b74727a860a56338e042c4420bb3f04b2fec5734175f4cb9fa853daf52b7e8

Download size:

5.29 GiB

Memory requirement (unknown):

Unknown — No exact process peak for this Qwen3.5 GGUF, recurrent-state and text workload.

System RAM / disk:

Unknown / Unknown

Placement:

single device. One CPU host; no tensor/pipeline sharding. CPU-only execution; --device none and zero GPU layers. No disk or accelerator offload.

KV cache:

F16 attention K/V cache; Gated DeltaNet recurrent state is separately runtime-managed, with no claimed state precision or full-cache byte estimate.

Workload:

4,096 token context; Unknown; Unknown; concurrency 1; batch 1; cache unknown; repetitions Unknown.

Tokenizer:

Qwen3.5-9B GGUF tokenizer with separately pinned Qwen native Jinja template · 3885219b6810b007914f3a7950a8d1b469d598a5; template c202236235762e1c871ad0ccb60c8ee5ba337b9a

Sampling / reasoning:

Qwen non-thinking general-task preset: temperature 0.7, top_p 0.8, top_k 20, min_p 0.0, presence_penalty 1.5, repeat_penalty 1.0. Validation max_tokens 512. Non-thinking text-only mode: native Qwen Jinja template, enable_thinking=false, --reasoning off. No /think or /nothink soft switch. Raw output mode; no tool parser, MTP drafter or vision projector.

Runtime prerequisites:

Ubuntu 24.04 x86_64; See prerequisites above.

Official artifact · License

Qwen publishes the base under Apache License 2.0; Unsloth’s derivative metadata declares Apache-2.0 and links that license. Read the full license and retain required license, copyright and attribution notices; mark modifications when redistributing covered work. Unsloth’s repository has no separate LICENSE file at this revision (404); attribution is based on its explicit metadata plus the linked Qwen license. Access is public/ungated as checked; no provider API subscription is required.

• qwen35-chat-template.jinja · 0 GiB · revision c202236235762e1c871ad0ccb60c8ee5ba337b9a

• Gated DeltaNet recurrent state, full-attention cache, weights, buffers and OS share CPU system RAM. GGUF file bytes alone do not prove fit.
• F16 cache flags describe attention K/V only; recurrent-state precision/overhead is not measured or inferred. No sibling or GPU throughput is transferred.

Install the runtime

1. 1. Build the pinned CPU server

   Install build dependencies and compile the exact official source revision. These packages are system shared dependencies; no CUDA/GPU driver is required. GGML_NATIVE=ON selects -march=native for this GNU/Linux x86 build, compiling for the host CPU. Build and run on the same machine; no AVX-512 requirement is imposed on the i5-12400.

   CommandCopy command

   sudo apt-get update
   sudo apt-get install -y build-essential cmake git curl ca-certificates openssl
   mkdir -p "$HOME/qwen35-9b-hosting/models"
   cd "$HOME/qwen35-9b-hosting"
   git clone https://github.com/ggml-org/llama.cpp.git
   cd llama.cpp
   git checkout --detach 620d2225cc436ab0b001e3200e590e3b286ef9ab
   cmake -S . -B build -DCMAKE_BUILD_TYPE=Release -DGGML_METAL=OFF -DGGML_NATIVE=ON
   cmake --build build --config Release -j 4
   ./build/bin/llama-server --version

   Expected result: Build succeeds and server version identifies pinned source. Stop if checkout or build fails.

Download the model

1. 1. Download the pinned Unsloth GGUF and Qwen template

   Verify both public immutable inputs. Qwen base/template and Unsloth quantization have separate revisions and authors. Use the explicit native Jinja template so non-thinking control does not depend on an uninspected embedded-template version. Stop on either hash mismatch.

   CommandCopy command

   cd "$HOME/qwen35-9b-hosting"
   curl --fail --location --output models/Qwen3.5-9B-Q4_K_M.gguf 'https://huggingface.co/unsloth/Qwen3.5-9B-GGUF/resolve/3885219b6810b007914f3a7950a8d1b469d598a5/Qwen3.5-9B-Q4_K_M.gguf'
   curl --fail --location --output models/qwen35-chat-template.jinja 'https://huggingface.co/Qwen/Qwen3.5-9B/raw/c202236235762e1c871ad0ccb60c8ee5ba337b9a/chat_template.jinja'
   sha256sum --check <<'SHA256'
   03b74727a860a56338e042c4420bb3f04b2fec5734175f4cb9fa853daf52b7e8  models/Qwen3.5-9B-Q4_K_M.gguf
   a4aee8afcf2e0711942cf848899be66016f8d14a889ff9ede07bca099c28f715  models/qwen35-chat-template.jinja
   SHA256

   Expected result: Both files report OK. GGUF metadata size 5,680,522,464 bytes; template 7756 bytes. No projector is downloaded.

Configure

1. 1. Create a private service credential

   Generate this only on your own host when following the guide. It is a local service password, not an OpenAI/provider credential. Keep it private and never paste it into tickets. Create it only when absent; preserve an existing key so a repeated setup does not silently rotate access. It is not created during this documentation run.

   CommandCopy command

   cd "$HOME/qwen35-9b-hosting"
   umask 077
   if [ ! -e .api-key ]; then
     openssl rand -hex 32 > .api-key
   fi
   test -s .api-key || { echo 'Existing .api-key is empty; restore it before continuing.'; exit 1; }
   chmod 600 .api-key

   Expected result: A non-empty private .api-key file exists; existing key preserved. No secret is printed.

Start the model

1. 1. Start one CPU-only sequence

   Run the foreground CPU server with the exact Qwen3.5 dense/hybrid architecture. Native template enable_thinking=false supplies a closed empty thinking prefix; --reasoning off makes this intent explicit. Raw-content mode does not promise separate reasoning_content. Use 4,096 total prompt/output context and one slot; reserve 512 output tokens. Qwen recommends at least 128K context for preserving thinking capabilities, so this 4K guide deliberately uses non-thinking general chat. No vision projector or image/video requests.

   CommandCopy command

   cd "$HOME/qwen35-9b-hosting"
   export QWEN35_9B_SERVICE_KEY="$(cat .api-key)"
   ./llama.cpp/build/bin/llama-server -m models/Qwen3.5-9B-Q4_K_M.gguf --alias qwen35-9b --host 127.0.0.1 --port 8080 --api-key "$QWEN35_9B_SERVICE_KEY" --device none -ngl 0 -t 4 -c 4096 -np 1 --cache-type-k f16 --cache-type-v f16 --jinja --chat-template-file models/qwen35-chat-template.jinja --chat-template-kwargs '{"enable_thinking":false}' --reasoning off --reasoning-format none --skip-chat-parsing --temp 0.7 --top-p 0.8 --top-k 20 --min-p 0.0 --presence-penalty 1.5 --repeat-penalty 1.0

   Expected result: Model initializes and listener starts. Pass health and authenticated chat checks before relying on the service. Command-line credentials may be visible to privileged/local users; trusted host required.

Verify the setup

1. 1. Check readiness and authenticated text generation

   Use a second server terminal and the private service key. Send only plain text. Check final response against the supplied source passage and verify finish_reason is not length; no exact wording is guaranteed. Do not expect reasoning_content in raw mode. If extended thinking appears, non-thinking setup validation failed.

   CommandCopy command

   cd "$HOME/qwen35-9b-hosting"
   export QWEN35_9B_SERVICE_KEY="$(cat .api-key)"
   curl --fail http://127.0.0.1:8080/health
   curl --fail http://127.0.0.1:8080/v1/chat/completions -H "Authorization: Bearer $QWEN35_9B_SERVICE_KEY" -H 'Content-Type: application/json' -d '{"model":"qwen35-9b","messages":[{"role":"user","content":"Summarize this supplied passage in one sentence without adding facts: The community library opens at 9am on weekdays and closes at 6pm. On Saturday it opens at 10am and closes at 4pm. It is closed on Sunday."}],"max_tokens":512,"temperature":0.7,"top_k":20,"top_p":0.8,"min_p":0.0,"presence_penalty":1.5,"repeat_penalty":1.0}'
   curl -o /dev/null -s -w '%{http_code}\n' http://127.0.0.1:8080/v1/chat/completions -H 'Content-Type: application/json' -d '{"model":"qwen35-9b","messages":[{"role":"user","content":"Hello"}]}'

   Expected result: Health HTTP 200 when ready; authenticated response contains non-empty choices[0].message.content with an accurate final summary and finish_reason not length, without a generated extended thinking trace. Unauthenticated chat returns HTTP 401. Verify facts yourself; output can be wrong.

Performance evidence

No compatible measurement available. Documented support and estimated memory do not establish response speed.

Limits and troubleshooting

• Documented configuration only. No weights, build, inference, full-fit, throughput or latency benchmark was executed by OpenTools.
• Unsloth Q4_K_M GGUF is 5,680,522,464 bytes. It is a derivative of Qwen/Qwen3.5-9B with separate quantizer revision; no equivalence to full precision quality is claimed.
• The native 9B model is a dense language backbone with a vision encoder, 32 layers and alternating three Gated DeltaNet layers per full-attention layer. Family-wide MoE marketing does not make this 9B checkpoint a sparse MoE. Attention KV and recurrent state must not be collapsed into a transformer-only cache formula.
• This guide is text-only. No mmproj/projector is provided and image/video requests are outside its configuration. The model’s multimodal capability is not a claim that this text recipe validates vision.
• Qwen thinks by default. This guide pins its native template and disables thinking via enable_thinking=false plus --reasoning off. /think and /nothink are not officially supported soft switches. A 4K non-thinking recipe does not preserve/validate full thinking capability; the publisher advises at least 128K context for that purpose.
• The publisher advertises 262,144 native context with extension options. This recipe configures 4,096 total prompt and output, one active sequence and 512 validation output tokens. Longer context, thinking, concurrent users and another cache policy create different configurations.
• Use Qwen’s non-thinking general-task sampler (temperature 0.7, top-p 0.8, top-k 20, min-p 0, presence penalty 1.5 and repetition penalty 1.0). The model card gives conflicting non-thinking reasoning presets in two sections; this recipe avoids that task/mode and uses the consistent general-task preset.
• Summarization validation checks a short text request only. It does not establish coding, multilingual, long-context, retrieval, agentic, structured-output or multimodal quality; generated answers can be wrong.
• Operating system packages and build tool dependencies resolve when installed. Save tool versions/build logs with any actual execution. Local hardware is Intel Core i5-12400 with four inference threads allocated; no exact-hardware benchmark is claimed.
• Loopback and a service key do not isolate the service from privileged local users. A shared/public service needs separately designed TLS, authentication and operations.

Download or checksum failure

Verify exact uppercase filename and pinned URL; remove only the incomplete dedicated model file and retry. Do not run a mismatching file.

Architecture unsupported or build fails

Confirm exact pinned source commit, Ubuntu compiler/CMake dependencies and GGUF. Do not substitute an older packaged llama.cpp silently.

Out of memory or context overflow

Stop server, free/provision RAM or reduce context and restart; shorten input/output within configured total. Allocation is not a measured guarantee; record any changed configuration.

Connection refused or HTTP 401

Check foreground server logs, 8080 binding, matching private key and Bearer header. Run the client on this same desktop and keep the loopback listener.

Unexpected thinking trace, template error or image request failure

Confirm both exact GGUF/template hashes, --jinja, enable_thinking=false and --reasoning off. Do not use /nothink as a substitute. Send only text; this recipe has no mmproj and does not support images/videos. If template or architecture loading fails, confirm exact pinned runtime and stop rather than substituting a different checkpoint.

Stop the model

1. 1. Stop the service

   Press Ctrl+C in server terminal. Unset key in each shell. Confirm this desktop localhost:8080 no longer serves the model.

   CommandCopy command

   unset QWEN35_9B_SERVICE_KEY

   Expected result: Server stops and localhost:8080 stops serving.

Uninstall

1. 1. Remove dedicated model, build and credential files

   Back up wanted logs/files first. Remove only this walkthrough directory after confirming its path. Shared apt packages can remain; do not remove other workloads.

   CommandCopy command

   rm -r -- "$HOME/qwen35-9b-hosting"

   Expected result: Dedicated files and local service credential removed; shared system dependencies remain.

Related guides

• https://opentools.ai/llms/qwen35-9b

Exact model and requirements

Artifact:

Unsloth (GGUF quantization); Qwen (base model) / Qwen3.5-9B Q4_K_M, Unsloth derivative · checkpoint unsloth/Qwen3.5-9B-GGUF

Publisher revision:

3885219b6810b007914f3a7950a8d1b469d598a5

Artifact digest:

sha256:03b74727a860a56338e042c4420bb3f04b2fec5734175f4cb9fa853daf52b7e8

Download size:

5.29 GiB

Memory requirement (unknown):

Unknown — No exact process peak for this Qwen3.5 GGUF, recurrent-state and text workload.

System RAM / disk:

Unknown / Unknown

Placement:

single device. One CPU host; no tensor/pipeline sharding. CPU-only execution; --device none and zero GPU layers. No disk or accelerator offload.

KV cache:

F16 attention K/V cache; Gated DeltaNet recurrent state is separately runtime-managed, with no claimed state precision or full-cache byte estimate.

Workload:

4,096 token context; Unknown; Unknown; concurrency 1; batch 1; cache unknown; repetitions Unknown.

Tokenizer:

Qwen3.5-9B GGUF tokenizer with separately pinned Qwen native Jinja template · 3885219b6810b007914f3a7950a8d1b469d598a5; template c202236235762e1c871ad0ccb60c8ee5ba337b9a

Sampling / reasoning:

Qwen non-thinking general-task preset: temperature 0.7, top_p 0.8, top_k 20, min_p 0.0, presence_penalty 1.5, repeat_penalty 1.0. Validation max_tokens 512. Non-thinking text-only mode: native Qwen Jinja template, enable_thinking=false, --reasoning off. No /think or /nothink soft switch. Raw output mode; no tool parser, MTP drafter or vision projector.

Runtime prerequisites:

Ubuntu 24.04 x86_64; See prerequisites above.

Official artifact · License

Qwen publishes the base under Apache License 2.0; Unsloth’s derivative metadata declares Apache-2.0 and links that license. Read the full license and retain required license, copyright and attribution notices; mark modifications when redistributing covered work. Unsloth’s repository has no separate LICENSE file at this revision (404); attribution is based on its explicit metadata plus the linked Qwen license. Access is public/ungated as checked; no provider API subscription is required.

• qwen35-chat-template.jinja · 0 GiB · revision c202236235762e1c871ad0ccb60c8ee5ba337b9a

• Gated DeltaNet recurrent state, full-attention cache, weights, buffers and OS share CPU system RAM. GGUF file bytes alone do not prove fit.
• F16 cache flags describe attention K/V only; recurrent-state precision/overhead is not measured or inferred. No sibling or GPU throughput is transferred.

Install the runtime

1. 1. Build the pinned CPU server

   Install build dependencies and compile the exact official source revision. These packages are system shared dependencies; no CUDA/GPU driver is required. GGML_NATIVE=ON selects -march=native for this GNU/Linux x86 build, compiling for the host CPU. Build and run on the same machine; the binary uses the instruction set exposed by this r7i.xlarge guest. Do not copy it to a different instance family without rebuilding.

   CommandCopy command

   sudo apt-get update
   sudo apt-get install -y build-essential cmake git curl ca-certificates openssl
   mkdir -p "$HOME/qwen35-9b-hosting/models"
   cd "$HOME/qwen35-9b-hosting"
   git clone https://github.com/ggml-org/llama.cpp.git
   cd llama.cpp
   git checkout --detach 620d2225cc436ab0b001e3200e590e3b286ef9ab
   cmake -S . -B build -DCMAKE_BUILD_TYPE=Release -DGGML_METAL=OFF -DGGML_NATIVE=ON
   cmake --build build --config Release -j 4
   ./build/bin/llama-server --version

   Expected result: Build succeeds and server version identifies pinned source. Stop if checkout or build fails.

Download the model

1. 1. Download the pinned Unsloth GGUF and Qwen template

   Verify both public immutable inputs. Qwen base/template and Unsloth quantization have separate revisions and authors. Use the explicit native Jinja template so non-thinking control does not depend on an uninspected embedded-template version. Stop on either hash mismatch.

   CommandCopy command

   cd "$HOME/qwen35-9b-hosting"
   curl --fail --location --output models/Qwen3.5-9B-Q4_K_M.gguf 'https://huggingface.co/unsloth/Qwen3.5-9B-GGUF/resolve/3885219b6810b007914f3a7950a8d1b469d598a5/Qwen3.5-9B-Q4_K_M.gguf'
   curl --fail --location --output models/qwen35-chat-template.jinja 'https://huggingface.co/Qwen/Qwen3.5-9B/raw/c202236235762e1c871ad0ccb60c8ee5ba337b9a/chat_template.jinja'
   sha256sum --check <<'SHA256'
   03b74727a860a56338e042c4420bb3f04b2fec5734175f4cb9fa853daf52b7e8  models/Qwen3.5-9B-Q4_K_M.gguf
   a4aee8afcf2e0711942cf848899be66016f8d14a889ff9ede07bca099c28f715  models/qwen35-chat-template.jinja
   SHA256

   Expected result: Both files report OK. GGUF metadata size 5,680,522,464 bytes; template 7756 bytes. No projector is downloaded.

Configure

1. 1. Create a private service credential

   Generate this only on your own host when following the guide. It is a local service password, not an OpenAI/provider credential. Keep it private and never paste it into tickets. Create it only when absent; preserve an existing key so a repeated setup does not silently rotate access. It is not created during this documentation run.

   CommandCopy command

   cd "$HOME/qwen35-9b-hosting"
   umask 077
   if [ ! -e .api-key ]; then
     openssl rand -hex 32 > .api-key
   fi
   test -s .api-key || { echo 'Existing .api-key is empty; restore it before continuing.'; exit 1; }
   chmod 600 .api-key

   Expected result: A non-empty private .api-key file exists; existing key preserved. No secret is printed.

Start the model

1. 1. Start one CPU-only sequence

   Run the foreground CPU server with the exact Qwen3.5 dense/hybrid architecture. Native template enable_thinking=false supplies a closed empty thinking prefix; --reasoning off makes this intent explicit. Raw-content mode does not promise separate reasoning_content. Use 4,096 total prompt/output context and one slot; reserve 512 output tokens. Qwen recommends at least 128K context for preserving thinking capabilities, so this 4K guide deliberately uses non-thinking general chat. No vision projector or image/video requests.

   CommandCopy command

   cd "$HOME/qwen35-9b-hosting"
   export QWEN35_9B_SERVICE_KEY="$(cat .api-key)"
   ./llama.cpp/build/bin/llama-server -m models/Qwen3.5-9B-Q4_K_M.gguf --alias qwen35-9b --host 127.0.0.1 --port 8080 --api-key "$QWEN35_9B_SERVICE_KEY" --device none -ngl 0 -t 4 -c 4096 -np 1 --cache-type-k f16 --cache-type-v f16 --jinja --chat-template-file models/qwen35-chat-template.jinja --chat-template-kwargs '{"enable_thinking":false}' --reasoning off --reasoning-format none --skip-chat-parsing --temp 0.7 --top-p 0.8 --top-k 20 --min-p 0.0 --presence-penalty 1.5 --repeat-penalty 1.0

   Expected result: Model initializes and listener starts. Pass health and authenticated chat checks before relying on the service. Command-line credentials may be visible to privileged/local users; trusted host required.

Verify the setup

1. 1. Check readiness and authenticated text generation

   Use a second server terminal and the private service key. Send only plain text. Check final response against the supplied source passage and verify finish_reason is not length; no exact wording is guaranteed. Do not expect reasoning_content in raw mode. If extended thinking appears, non-thinking setup validation failed.

   CommandCopy command

   cd "$HOME/qwen35-9b-hosting"
   export QWEN35_9B_SERVICE_KEY="$(cat .api-key)"
   curl --fail http://127.0.0.1:8080/health
   curl --fail http://127.0.0.1:8080/v1/chat/completions -H "Authorization: Bearer $QWEN35_9B_SERVICE_KEY" -H 'Content-Type: application/json' -d '{"model":"qwen35-9b","messages":[{"role":"user","content":"Summarize this supplied passage in one sentence without adding facts: The community library opens at 9am on weekdays and closes at 6pm. On Saturday it opens at 10am and closes at 4pm. It is closed on Sunday."}],"max_tokens":512,"temperature":0.7,"top_k":20,"top_p":0.8,"min_p":0.0,"presence_penalty":1.5,"repeat_penalty":1.0}'
   curl -o /dev/null -s -w '%{http_code}\n' http://127.0.0.1:8080/v1/chat/completions -H 'Content-Type: application/json' -d '{"model":"qwen35-9b","messages":[{"role":"user","content":"Hello"}]}'

   Expected result: Health HTTP 200 when ready; authenticated response contains non-empty choices[0].message.content with an accurate final summary and finish_reason not length, without a generated extended thinking trace. Unauthenticated chat returns HTTP 401. Verify facts yourself; output can be wrong.

Performance evidence

No compatible measurement available. Documented support and estimated memory do not establish response speed.

Limits and troubleshooting

• Documented configuration only. No weights, build, inference, full-fit, throughput or latency benchmark was executed by OpenTools.
• Unsloth Q4_K_M GGUF is 5,680,522,464 bytes. It is a derivative of Qwen/Qwen3.5-9B with separate quantizer revision; no equivalence to full precision quality is claimed.
• The native 9B model is a dense language backbone with a vision encoder, 32 layers and alternating three Gated DeltaNet layers per full-attention layer. Family-wide MoE marketing does not make this 9B checkpoint a sparse MoE. Attention KV and recurrent state must not be collapsed into a transformer-only cache formula.
• This guide is text-only. No mmproj/projector is provided and image/video requests are outside its configuration. The model’s multimodal capability is not a claim that this text recipe validates vision.
• Qwen thinks by default. This guide pins its native template and disables thinking via enable_thinking=false plus --reasoning off. /think and /nothink are not officially supported soft switches. A 4K non-thinking recipe does not preserve/validate full thinking capability; the publisher advises at least 128K context for that purpose.
• The publisher advertises 262,144 native context with extension options. This recipe configures 4,096 total prompt and output, one active sequence and 512 validation output tokens. Longer context, thinking, concurrent users and another cache policy create different configurations.
• Use Qwen’s non-thinking general-task sampler (temperature 0.7, top-p 0.8, top-k 20, min-p 0, presence penalty 1.5 and repetition penalty 1.0). The model card gives conflicting non-thinking reasoning presets in two sections; this recipe avoids that task/mode and uses the consistent general-task preset.
• Summarization validation checks a short text request only. It does not establish coding, multilingual, long-context, retrieval, agentic, structured-output or multimodal quality; generated answers can be wrong.
• Operating system packages and build tool dependencies resolve when installed. Save tool versions/build logs with any actual execution. Cloud hardware is AWS r7i.xlarge with four vCPUs and 32 GiB host RAM; four inference threads are allocated. No exact-instance benchmark is claimed.
• Loopback and a service key do not isolate the service from privileged local users. A shared/public service needs separately designed TLS, authentication and operations.

Download or checksum failure

Verify exact uppercase filename and pinned URL; remove only the incomplete dedicated model file and retry. Do not run a mismatching file.

Architecture unsupported or build fails

Confirm exact pinned source commit, Ubuntu compiler/CMake dependencies and GGUF. Do not substitute an older packaged llama.cpp silently.

Out of memory or context overflow

Stop server, free/provision RAM or reduce context and restart; shorten input/output within configured total. Allocation is not a measured guarantee; record any changed configuration.

Connection refused or HTTP 401

Check foreground server logs, 8080 binding, matching private key and Bearer header. Cloud client must have working SSH tunnel to the correct VM; do not expose the port to repair a tunnel.

Unexpected thinking trace, template error or image request failure

Confirm both exact GGUF/template hashes, --jinja, enable_thinking=false and --reasoning off. Do not use /nothink as a substitute. Send only text; this recipe has no mmproj and does not support images/videos. If template or architecture loading fails, confirm exact pinned runtime and stop rather than substituting a different checkpoint.

Stop the model

1. 1. Stop the service

   Press Ctrl+C in server terminal. Unset key in each shell. Close the SSH tunnel with Ctrl+C. Stopping the process alone does not stop EC2 billing.

   CommandCopy command

   unset QWEN35_9B_SERVICE_KEY

   Expected result: Server stops and localhost:8080 stops serving.

Uninstall

1. 1. Remove dedicated model, build and credential files

   Back up wanted logs/files first. Remove only this walkthrough directory after confirming its path. Shared apt packages can remain; do not remove other workloads.

   CommandCopy command

   rm -r -- "$HOME/qwen35-9b-hosting"

   Expected result: Dedicated files and local service credential removed; shared system dependencies remain.

Shut down cloud resources

1. 1. End cloud billing resources

   In EC2 us-east-1, stop for later reuse or terminate this exact VM. Review Delete on termination for its root volume. Delete unwanted detached EBS volumes, snapshots and reserved/Elastic/public IP resources. Stopped VM storage and retained resources may still bill. Check Bills/Cost Explorer later for final usage.

   Expected result: Exact instance is stopped/terminated as intended; no unwanted attached/retained billed resources remain. Instance compute stops only after its lifecycle transition, not when llama-server stops.

Related guides

• https://opentools.ai/llms/qwen35-9b

Exact model and requirements

Artifact:

Unsloth (GGUF quantization); Qwen (base model) / Qwen3.5-9B Q4_K_M, Unsloth derivative · checkpoint unsloth/Qwen3.5-9B-GGUF

Publisher revision:

3885219b6810b007914f3a7950a8d1b469d598a5

Artifact digest:

sha256:03b74727a860a56338e042c4420bb3f04b2fec5734175f4cb9fa853daf52b7e8

Download size:

5.29 GiB

Memory requirement (unknown):

Unknown — No exact process peak for this Qwen3.5 GGUF, recurrent-state and text workload.

System RAM / disk:

Unknown / Unknown

Placement:

single device. One CPU host; no tensor/pipeline sharding. CPU-only execution; --device none and zero GPU layers. No disk or accelerator offload. Vision projector also explicitly CPU-only via --no-mmproj-offload.

KV cache:

F16 attention K/V cache; Gated DeltaNet recurrent state is separately runtime-managed, with no claimed state precision or full-cache byte estimate.

Workload:

4,096 token context; Unknown; Unknown; concurrency 1; batch 1; cache unknown; repetitions Unknown.

Tokenizer:

Qwen3.5-9B GGUF tokenizer with separately pinned Qwen native Jinja template · 3885219b6810b007914f3a7950a8d1b469d598a5; template c202236235762e1c871ad0ccb60c8ee5ba337b9a

Sampling / reasoning:

Qwen non-thinking general-task preset: temperature 0.7, top_p 0.8, top_k 20, min_p 0.0, presence_penalty 1.5, repeat_penalty 1.0. Validation max_tokens 512. Non-thinking single-image mode: native Qwen Jinja template, enable_thinking=false, --reasoning off. No /think or /nothink soft switch. Raw output mode; no tool parser, MTP drafter. Single image input through pinned QWEN3VL vision path; enable_thinking=false also applies to multimodal messages in the native 9B template.

Runtime prerequisites:

Ubuntu 24.04 x86_64; See prerequisites above.

Official artifact · License

Qwen publishes the base under Apache License 2.0; Unsloth’s derivative metadata declares Apache-2.0 and links that license. Read the full license and retain required license, copyright and attribution notices; mark modifications when redistributing covered work. Unsloth’s repository has no separate LICENSE file at this revision (404); attribution is based on its explicit metadata plus the linked Qwen license. Access is public/ungated as checked; no provider API subscription is required.

• qwen35-chat-template.jinja · 0 GiB · revision c202236235762e1c871ad0ccb60c8ee5ba337b9a
• mmproj-F16.gguf · 0.86 GiB · revision 3885219b6810b007914f3a7950a8d1b469d598a5

• Gated DeltaNet recurrent state, full-attention cache, weights, buffers and OS share CPU system RAM. GGUF file bytes alone do not prove fit.
• F16 cache flags describe attention K/V only; recurrent-state precision/overhead is not measured or inferred. No sibling or GPU throughput is transferred.
• Additional 918166080-byte F16 projector, decoded pixels, vision encoder, image embeddings and warmup buffers require host RAM beyond text-only allocations. No fit, process peak or vision-token measurement exists. 32 GiB host allocation is a walkthrough choice, not a validated minimum.

Install the runtime

1. 1. Build the pinned CPU server

   Install build dependencies and compile the exact official source revision. These packages are system shared dependencies; no CUDA/GPU driver is required. GGML_NATIVE=ON selects -march=native for this GNU/Linux x86 build, compiling for the host CPU. Build and run on the same machine; no AVX-512 requirement is imposed on the i5-12400.

   CommandCopy command

   sudo apt-get update
   sudo apt-get install -y build-essential cmake git curl ca-certificates openssl python3
   mkdir -p "$HOME/qwen35-9b-vision-hosting/models"
   cd "$HOME/qwen35-9b-vision-hosting"
   git clone https://github.com/ggml-org/llama.cpp.git
   cd llama.cpp
   git checkout --detach 620d2225cc436ab0b001e3200e590e3b286ef9ab
   cmake -S . -B build -DCMAKE_BUILD_TYPE=Release -DGGML_METAL=OFF -DGGML_NATIVE=ON
   cmake --build build --config Release -j 4
   ./build/bin/llama-server --version

   Expected result: Build succeeds and server version identifies pinned source. Stop if checkout or build fails.

Download the model

1. 1. Download language weights, native template, matching projector and tiny image fixture

   Verify both public immutable inputs. Qwen base/template and Unsloth quantization have separate revisions and authors. Use the explicit native Jinja template so non-thinking control does not depend on an uninspected embedded-template version. Stop on either hash mismatch. The additional projector is Unsloth-published for this exact 9B derivative repository. The official runtime test fixture is for a private reproducibility check; no redistribution rights to the newspaper photograph are asserted.

   CommandCopy command

   cd "$HOME/qwen35-9b-vision-hosting"
   curl --fail --location --output models/Qwen3.5-9B-Q4_K_M.gguf 'https://huggingface.co/unsloth/Qwen3.5-9B-GGUF/resolve/3885219b6810b007914f3a7950a8d1b469d598a5/Qwen3.5-9B-Q4_K_M.gguf'
   curl --fail --location --output models/qwen35-chat-template.jinja 'https://huggingface.co/Qwen/Qwen3.5-9B/raw/c202236235762e1c871ad0ccb60c8ee5ba337b9a/chat_template.jinja'
   sha256sum --check <<'SHA256'
   03b74727a860a56338e042c4420bb3f04b2fec5734175f4cb9fa853daf52b7e8  models/Qwen3.5-9B-Q4_K_M.gguf
   a4aee8afcf2e0711942cf848899be66016f8d14a889ff9ede07bca099c28f715  models/qwen35-chat-template.jinja
   SHA256
   curl --fail --location --output models/mmproj-F16.gguf 'https://huggingface.co/unsloth/Qwen3.5-9B-GGUF/resolve/3885219b6810b007914f3a7950a8d1b469d598a5/mmproj-F16.gguf'
   curl --fail --location --output fixture.jpg 'https://raw.githubusercontent.com/ggml-org/llama.cpp/620d2225cc436ab0b001e3200e590e3b286ef9ab/tools/mtmd/test-1.jpeg'
   sha256sum --check <<'SHA256'
   f70dc3509053962b0d0d3ee8a7eacebf5d60aa560cad78254ae8698516ae029f  models/mmproj-F16.gguf
   2dff664c0c8aaea18aff8cbe7e868845b775e90cdd7a0bac98df709b131deaa3  fixture.jpg
   SHA256

   Expected result: All four immutable files report OK. Projector 918166080 bytes; fixture 124071 bytes. Stop on mismatch.

Configure

1. 1. Create a private service credential

   Generate this only on your own host when following the guide. It is a local service password, not an OpenAI/provider credential. Keep it private and never paste it into tickets. Create it only when absent; preserve an existing key so a repeated setup does not silently rotate access. It is not created during this documentation run.

   CommandCopy command

   cd "$HOME/qwen35-9b-vision-hosting"
   umask 077
   if [ ! -e .api-key ]; then
     openssl rand -hex 32 > .api-key
   fi
   test -s .api-key || { echo 'Existing .api-key is empty; restore it before continuing.'; exit 1; }
   chmod 600 .api-key

   Expected result: A non-empty private .api-key file exists; existing key preserved. No secret is printed.

Start the model

1. 1. Start one CPU-only sequence

   Start the exact 9B language model with matching QWEN3VL vision projector. The pinned converter explicitly maps Qwen3_5ForConditionalGeneration and Qwen/Qwen3.5-9B to QWEN3VL; mtmd selects dynamic-resolution preprocessing, clip uses bicubic resize and spatial merge metadata. --image-min-tokens 1024 and --image-max-tokens 1024 bound this walkthrough image encoding; actual emitted token count remains unmeasured. Image/template/text/output all share 4096 context. Keep 512 output tokens. Non-thinking general-task sampler is the 9B publisher preset and its native template handles image parts with enable_thinking=false. Not a grounding benchmark.

   CommandCopy command

   cd "$HOME/qwen35-9b-vision-hosting"
   export QWEN35_9B_SERVICE_KEY="$(cat .api-key)"
   ./llama.cpp/build/bin/llama-server -m models/Qwen3.5-9B-Q4_K_M.gguf --mmproj models/mmproj-F16.gguf --no-mmproj-offload --image-min-tokens 1024 --image-max-tokens 1024 --alias qwen35-9b --host 127.0.0.1 --port 8080 --api-key "$QWEN35_9B_SERVICE_KEY" --device none -ngl 0 -t 4 -c 4096 -np 1 --cache-type-k f16 --cache-type-v f16 --jinja --chat-template-file models/qwen35-chat-template.jinja --chat-template-kwargs '{"enable_thinking":false}' --reasoning off --reasoning-format none --skip-chat-parsing --temp 0.7 --top-p 0.8 --top-k 20 --min-p 0.0 --presence-penalty 1.5 --repeat-penalty 1.0

   Expected result: Model initializes and listener starts. Pass health and authenticated chat checks before relying on the service. Command-line credentials may be visible to privileged/local users; trusted host required.

Verify the setup

1. 1. Check readiness and authenticated text generation

   Use a second server terminal and the private service key. Send only plain text. Check final response against the supplied source passage and verify finish_reason is not length; no exact wording is guaranteed. Do not expect reasoning_content in raw mode. If extended thinking appears, non-thinking setup validation failed.

   CommandCopy command

   cd "$HOME/qwen35-9b-vision-hosting"
   export QWEN35_9B_SERVICE_KEY="$(cat .api-key)"
   curl --fail http://127.0.0.1:8080/health
   curl --fail http://127.0.0.1:8080/v1/chat/completions -H "Authorization: Bearer $QWEN35_9B_SERVICE_KEY" -H 'Content-Type: application/json' -d '{"model":"qwen35-9b","messages":[{"role":"user","content":"Summarize this supplied passage in one sentence without adding facts: The community library opens at 9am on weekdays and closes at 6pm. On Saturday it opens at 10am and closes at 4pm. It is closed on Sunday."}],"max_tokens":512,"temperature":0.7,"top_k":20,"top_p":0.8,"min_p":0.0,"presence_penalty":1.5,"repeat_penalty":1.0}'
   curl -o /dev/null -s -w '%{http_code}\n' http://127.0.0.1:8080/v1/chat/completions -H 'Content-Type: application/json' -d '{"model":"qwen35-9b","messages":[{"role":"user","content":"Hello"}]}'

   Expected result: Health HTTP 200 when ready; authenticated response contains non-empty choices[0].message.content with an accurate final summary and finish_reason not length, without a generated extended thinking trace. Unauthenticated chat returns HTTP 401. Verify facts yourself; output can be wrong.

2. 2. Check authenticated single-image understanding

   Run on the same server host (for cloud, inside the SSH shell). The JSON content array combines text and image_url with a local file encoded as data:image/jpeg;base64; no server-side remote image fetch or media-path flag is needed. Inspect content and finish_reason yourself. The fixture visibly shows a newspaper front page: The New York Times, MEN WALK ON MOON, astronauts landing/collecting rocks/planting a flag. A useful answer identifies the newspaper and the moon-landing headline; wording is not prescribed. Do not use unseen details or exact token counts as pass criteria.

   CommandCopy command

   cd "$HOME/qwen35-9b-vision-hosting"
   python3 - <<'PYREQUEST'
   import base64,json,pathlib
   image=base64.b64encode(pathlib.Path('fixture.jpg').read_bytes()).decode()
   request={'model':'qwen35-9b','messages':[{'role':'user','content':[{'type':'text','text':'Identify the newspaper and transcribe its largest headline. Then briefly describe the front page.'},{'type':'image_url','image_url':{'url':'data:image/jpeg;base64,'+image}}]}],'max_tokens':512,'temperature':0.7,'top_p':0.8,'top_k':20,'min_p':0.0,'presence_penalty':1.5,'repeat_penalty':1.0}
   pathlib.Path('image-request.json').write_text(json.dumps(request))
   PYREQUEST
   curl --fail http://127.0.0.1:8080/v1/chat/completions -H "Authorization: Bearer $QWEN35_9B_SERVICE_KEY" -H 'Content-Type: application/json' --data-binary @image-request.json

   Expected result: Authenticated HTTP 200 with nonempty choices[0].message.content identifying The New York Times and MEN WALK ON MOON; inspect grounded accuracy and finish_reason not length. No exact generated response promised. Image rejection, fabricated content, truncation or extended thinking fails this smoke check.

Performance evidence

No compatible measurement available. Documented support and estimated memory do not establish response speed.

Limits and troubleshooting

• Documented configuration only. No weights, build, inference, full-fit, throughput or latency benchmark was executed by OpenTools.
• Unsloth Q4_K_M GGUF is 5,680,522,464 bytes. It is a derivative of Qwen/Qwen3.5-9B with separate quantizer revision; no equivalence to full precision quality is claimed.
• The native 9B model is a dense language backbone with a vision encoder, 32 layers and alternating three Gated DeltaNet layers per full-attention layer. Family-wide MoE marketing does not make this 9B checkpoint a sparse MoE. Attention KV and recurrent state must not be collapsed into a transformer-only cache formula.
• Qwen thinks by default. This guide pins its native template and disables thinking via enable_thinking=false plus --reasoning off. /think and /nothink are not officially supported soft switches. A 4K non-thinking recipe does not preserve/validate full thinking capability; the publisher advises at least 128K context for that purpose.
• The publisher advertises 262,144 native context with extension options. This recipe configures 4,096 total prompt and output, one active sequence and 512 validation output tokens. Longer context, thinking, concurrent users and another cache policy create different configurations.
• Use Qwen’s non-thinking general-task sampler (temperature 0.7, top-p 0.8, top-k 20, min-p 0, presence penalty 1.5 and repetition penalty 1.0). The model card gives conflicting non-thinking reasoning presets in two sections; this recipe avoids that task/mode and uses the consistent general-task preset.
• Operating system packages and build tool dependencies resolve when installed. Save tool versions/build logs with any actual execution. Local hardware is Intel Core i5-12400 with four inference threads allocated; no exact-hardware benchmark is claimed.
• Loopback and a service key do not isolate the service from privileged local users. A shared/public service needs separately designed TLS, authentication and operations.
• Single-image description and the inherited supplied-passage text check are smoke checks only, not measured visual/OCR/grounding quality. The server may warm up larger vision buffers than the request; file bytes and the image-token ceiling do not prove fit. Actual prompt/image tokens, encoder memory, speed and quality remain unknown.
• No image preprocessing is delegated to Transformers: libmtmd uses its pinned dynamic-resolution QWEN3VL path, bicubic resize, model spatial merge metadata and CLI image token bounds. Do not transplant Hugging Face min_pixels/max_pixels request keys; this recipe sets server flags.

Download or checksum failure

Verify exact uppercase filename and pinned URL; remove only the incomplete dedicated model file and retry. Do not run a mismatching file.

Architecture unsupported or build fails

Confirm exact pinned source commit, Ubuntu compiler/CMake dependencies and GGUF. Do not substitute an older packaged llama.cpp silently.

Out of memory or context overflow

Stop server, free/provision RAM or reduce context and restart; shorten input/output within configured total. Allocation is not a measured guarantee; record any changed configuration.

Connection refused or HTTP 401

Check foreground server logs, 8080 binding, matching private key and Bearer header. Run the client on this same desktop and keep the loopback listener.

Unexpected thinking, template or projector failure

Check exact model/template/projector hashes, native Jinja enable_thinking=false, --reasoning off and --no-mmproj-offload. Confirm image_url content array and data URI. Stop on embedding/architecture mismatch; do not substitute another size projector.

Stop the model

1. 1. Stop the service

   Press Ctrl+C in server terminal. Unset key in each shell. Confirm this desktop localhost:8080 no longer serves the model.

   CommandCopy command

   unset QWEN35_9B_SERVICE_KEY

   Expected result: Server stops and localhost:8080 stops serving.

Uninstall

1. 1. Remove dedicated model, build and credential files

   Back up wanted logs/files first. Remove only this walkthrough directory after confirming its path. Shared apt packages can remain; do not remove other workloads.

   CommandCopy command

   rm -r -- "$HOME/qwen35-9b-vision-hosting"

   Expected result: Dedicated files and local service credential removed; shared system dependencies remain.

Related guides

• https://opentools.ai/llms/qwen35-9b

Exact model and requirements

Artifact:

Unsloth (GGUF quantization); Qwen (base model) / Qwen3.5-9B Q4_K_M, Unsloth derivative · checkpoint unsloth/Qwen3.5-9B-GGUF

Publisher revision:

3885219b6810b007914f3a7950a8d1b469d598a5

Artifact digest:

sha256:03b74727a860a56338e042c4420bb3f04b2fec5734175f4cb9fa853daf52b7e8

Download size:

5.29 GiB

Memory requirement (unknown):

Unknown — No exact process peak for this Qwen3.5 GGUF, recurrent-state and text workload.

System RAM / disk:

Unknown / Unknown

Placement:

single device. One CPU host; no tensor/pipeline sharding. CPU-only execution; --device none and zero GPU layers. No disk or accelerator offload. Vision projector also explicitly CPU-only via --no-mmproj-offload.

KV cache:

F16 attention K/V cache; Gated DeltaNet recurrent state is separately runtime-managed, with no claimed state precision or full-cache byte estimate.

Workload:

4,096 token context; Unknown; Unknown; concurrency 1; batch 1; cache unknown; repetitions Unknown.

Tokenizer:

Qwen3.5-9B GGUF tokenizer with separately pinned Qwen native Jinja template · 3885219b6810b007914f3a7950a8d1b469d598a5; template c202236235762e1c871ad0ccb60c8ee5ba337b9a

Sampling / reasoning:

Qwen non-thinking general-task preset: temperature 0.7, top_p 0.8, top_k 20, min_p 0.0, presence_penalty 1.5, repeat_penalty 1.0. Validation max_tokens 512. Non-thinking single-image mode: native Qwen Jinja template, enable_thinking=false, --reasoning off. No /think or /nothink soft switch. Raw output mode; no tool parser, MTP drafter. Single image input through pinned QWEN3VL vision path; enable_thinking=false also applies to multimodal messages in the native 9B template.

Runtime prerequisites:

Ubuntu 24.04 x86_64; See prerequisites above.

Official artifact · License

Qwen publishes the base under Apache License 2.0; Unsloth’s derivative metadata declares Apache-2.0 and links that license. Read the full license and retain required license, copyright and attribution notices; mark modifications when redistributing covered work. Unsloth’s repository has no separate LICENSE file at this revision (404); attribution is based on its explicit metadata plus the linked Qwen license. Access is public/ungated as checked; no provider API subscription is required.

• qwen35-chat-template.jinja · 0 GiB · revision c202236235762e1c871ad0ccb60c8ee5ba337b9a
• mmproj-F16.gguf · 0.86 GiB · revision 3885219b6810b007914f3a7950a8d1b469d598a5

• Gated DeltaNet recurrent state, full-attention cache, weights, buffers and OS share CPU system RAM. GGUF file bytes alone do not prove fit.
• F16 cache flags describe attention K/V only; recurrent-state precision/overhead is not measured or inferred. No sibling or GPU throughput is transferred.
• Additional 918166080-byte F16 projector, decoded pixels, vision encoder, image embeddings and warmup buffers require host RAM beyond text-only allocations. No fit, process peak or vision-token measurement exists. 32 GiB host allocation is a walkthrough choice, not a validated minimum.

Install the runtime

1. 1. Build the pinned CPU server

   Install build dependencies and compile the exact official source revision. These packages are system shared dependencies; no CUDA/GPU driver is required. GGML_NATIVE=ON selects -march=native for this GNU/Linux x86 build, compiling for the host CPU. Build and run on the same machine; the binary uses the instruction set exposed by this r7i.xlarge guest. Do not copy it to a different instance family without rebuilding.

   CommandCopy command

   sudo apt-get update
   sudo apt-get install -y build-essential cmake git curl ca-certificates openssl python3
   mkdir -p "$HOME/qwen35-9b-vision-hosting/models"
   cd "$HOME/qwen35-9b-vision-hosting"
   git clone https://github.com/ggml-org/llama.cpp.git
   cd llama.cpp
   git checkout --detach 620d2225cc436ab0b001e3200e590e3b286ef9ab
   cmake -S . -B build -DCMAKE_BUILD_TYPE=Release -DGGML_METAL=OFF -DGGML_NATIVE=ON
   cmake --build build --config Release -j 4
   ./build/bin/llama-server --version

   Expected result: Build succeeds and server version identifies pinned source. Stop if checkout or build fails.

Download the model

1. 1. Download language weights, native template, matching projector and tiny image fixture

   Verify both public immutable inputs. Qwen base/template and Unsloth quantization have separate revisions and authors. Use the explicit native Jinja template so non-thinking control does not depend on an uninspected embedded-template version. Stop on either hash mismatch. The additional projector is Unsloth-published for this exact 9B derivative repository. The official runtime test fixture is for a private reproducibility check; no redistribution rights to the newspaper photograph are asserted.

   CommandCopy command

   cd "$HOME/qwen35-9b-vision-hosting"
   curl --fail --location --output models/Qwen3.5-9B-Q4_K_M.gguf 'https://huggingface.co/unsloth/Qwen3.5-9B-GGUF/resolve/3885219b6810b007914f3a7950a8d1b469d598a5/Qwen3.5-9B-Q4_K_M.gguf'
   curl --fail --location --output models/qwen35-chat-template.jinja 'https://huggingface.co/Qwen/Qwen3.5-9B/raw/c202236235762e1c871ad0ccb60c8ee5ba337b9a/chat_template.jinja'
   sha256sum --check <<'SHA256'
   03b74727a860a56338e042c4420bb3f04b2fec5734175f4cb9fa853daf52b7e8  models/Qwen3.5-9B-Q4_K_M.gguf
   a4aee8afcf2e0711942cf848899be66016f8d14a889ff9ede07bca099c28f715  models/qwen35-chat-template.jinja
   SHA256
   curl --fail --location --output models/mmproj-F16.gguf 'https://huggingface.co/unsloth/Qwen3.5-9B-GGUF/resolve/3885219b6810b007914f3a7950a8d1b469d598a5/mmproj-F16.gguf'
   curl --fail --location --output fixture.jpg 'https://raw.githubusercontent.com/ggml-org/llama.cpp/620d2225cc436ab0b001e3200e590e3b286ef9ab/tools/mtmd/test-1.jpeg'
   sha256sum --check <<'SHA256'
   f70dc3509053962b0d0d3ee8a7eacebf5d60aa560cad78254ae8698516ae029f  models/mmproj-F16.gguf
   2dff664c0c8aaea18aff8cbe7e868845b775e90cdd7a0bac98df709b131deaa3  fixture.jpg
   SHA256

   Expected result: All four immutable files report OK. Projector 918166080 bytes; fixture 124071 bytes. Stop on mismatch.

Configure

1. 1. Create a private service credential

   Generate this only on your own host when following the guide. It is a local service password, not an OpenAI/provider credential. Keep it private and never paste it into tickets. Create it only when absent; preserve an existing key so a repeated setup does not silently rotate access. It is not created during this documentation run.

   CommandCopy command

   cd "$HOME/qwen35-9b-vision-hosting"
   umask 077
   if [ ! -e .api-key ]; then
     openssl rand -hex 32 > .api-key
   fi
   test -s .api-key || { echo 'Existing .api-key is empty; restore it before continuing.'; exit 1; }
   chmod 600 .api-key

   Expected result: A non-empty private .api-key file exists; existing key preserved. No secret is printed.

Start the model

1. 1. Start one CPU-only sequence

   Start the exact 9B language model with matching QWEN3VL vision projector. The pinned converter explicitly maps Qwen3_5ForConditionalGeneration and Qwen/Qwen3.5-9B to QWEN3VL; mtmd selects dynamic-resolution preprocessing, clip uses bicubic resize and spatial merge metadata. --image-min-tokens 1024 and --image-max-tokens 1024 bound this walkthrough image encoding; actual emitted token count remains unmeasured. Image/template/text/output all share 4096 context. Keep 512 output tokens. Non-thinking general-task sampler is the 9B publisher preset and its native template handles image parts with enable_thinking=false. Not a grounding benchmark.

   CommandCopy command

   cd "$HOME/qwen35-9b-vision-hosting"
   export QWEN35_9B_SERVICE_KEY="$(cat .api-key)"
   ./llama.cpp/build/bin/llama-server -m models/Qwen3.5-9B-Q4_K_M.gguf --mmproj models/mmproj-F16.gguf --no-mmproj-offload --image-min-tokens 1024 --image-max-tokens 1024 --alias qwen35-9b --host 127.0.0.1 --port 8080 --api-key "$QWEN35_9B_SERVICE_KEY" --device none -ngl 0 -t 4 -c 4096 -np 1 --cache-type-k f16 --cache-type-v f16 --jinja --chat-template-file models/qwen35-chat-template.jinja --chat-template-kwargs '{"enable_thinking":false}' --reasoning off --reasoning-format none --skip-chat-parsing --temp 0.7 --top-p 0.8 --top-k 20 --min-p 0.0 --presence-penalty 1.5 --repeat-penalty 1.0

   Expected result: Model initializes and listener starts. Pass health and authenticated chat checks before relying on the service. Command-line credentials may be visible to privileged/local users; trusted host required.

Verify the setup

1. 1. Check readiness and authenticated text generation

   Use a second server terminal and the private service key. Send only plain text. Check final response against the supplied source passage and verify finish_reason is not length; no exact wording is guaranteed. Do not expect reasoning_content in raw mode. If extended thinking appears, non-thinking setup validation failed.

   CommandCopy command

   cd "$HOME/qwen35-9b-vision-hosting"
   export QWEN35_9B_SERVICE_KEY="$(cat .api-key)"
   curl --fail http://127.0.0.1:8080/health
   curl --fail http://127.0.0.1:8080/v1/chat/completions -H "Authorization: Bearer $QWEN35_9B_SERVICE_KEY" -H 'Content-Type: application/json' -d '{"model":"qwen35-9b","messages":[{"role":"user","content":"Summarize this supplied passage in one sentence without adding facts: The community library opens at 9am on weekdays and closes at 6pm. On Saturday it opens at 10am and closes at 4pm. It is closed on Sunday."}],"max_tokens":512,"temperature":0.7,"top_k":20,"top_p":0.8,"min_p":0.0,"presence_penalty":1.5,"repeat_penalty":1.0}'
   curl -o /dev/null -s -w '%{http_code}\n' http://127.0.0.1:8080/v1/chat/completions -H 'Content-Type: application/json' -d '{"model":"qwen35-9b","messages":[{"role":"user","content":"Hello"}]}'

   Expected result: Health HTTP 200 when ready; authenticated response contains non-empty choices[0].message.content with an accurate final summary and finish_reason not length, without a generated extended thinking trace. Unauthenticated chat returns HTTP 401. Verify facts yourself; output can be wrong.

2. 2. Check authenticated single-image understanding

   Run on the same server host (for cloud, inside the SSH shell). The JSON content array combines text and image_url with a local file encoded as data:image/jpeg;base64; no server-side remote image fetch or media-path flag is needed. Inspect content and finish_reason yourself. The fixture visibly shows a newspaper front page: The New York Times, MEN WALK ON MOON, astronauts landing/collecting rocks/planting a flag. A useful answer identifies the newspaper and the moon-landing headline; wording is not prescribed. Do not use unseen details or exact token counts as pass criteria.

   CommandCopy command

   cd "$HOME/qwen35-9b-vision-hosting"
   python3 - <<'PYREQUEST'
   import base64,json,pathlib
   image=base64.b64encode(pathlib.Path('fixture.jpg').read_bytes()).decode()
   request={'model':'qwen35-9b','messages':[{'role':'user','content':[{'type':'text','text':'Identify the newspaper and transcribe its largest headline. Then briefly describe the front page.'},{'type':'image_url','image_url':{'url':'data:image/jpeg;base64,'+image}}]}],'max_tokens':512,'temperature':0.7,'top_p':0.8,'top_k':20,'min_p':0.0,'presence_penalty':1.5,'repeat_penalty':1.0}
   pathlib.Path('image-request.json').write_text(json.dumps(request))
   PYREQUEST
   curl --fail http://127.0.0.1:8080/v1/chat/completions -H "Authorization: Bearer $QWEN35_9B_SERVICE_KEY" -H 'Content-Type: application/json' --data-binary @image-request.json

   Expected result: Authenticated HTTP 200 with nonempty choices[0].message.content identifying The New York Times and MEN WALK ON MOON; inspect grounded accuracy and finish_reason not length. No exact generated response promised. Image rejection, fabricated content, truncation or extended thinking fails this smoke check.

Performance evidence

No compatible measurement available. Documented support and estimated memory do not establish response speed.

Limits and troubleshooting

• Documented configuration only. No weights, build, inference, full-fit, throughput or latency benchmark was executed by OpenTools.
• Unsloth Q4_K_M GGUF is 5,680,522,464 bytes. It is a derivative of Qwen/Qwen3.5-9B with separate quantizer revision; no equivalence to full precision quality is claimed.
• The native 9B model is a dense language backbone with a vision encoder, 32 layers and alternating three Gated DeltaNet layers per full-attention layer. Family-wide MoE marketing does not make this 9B checkpoint a sparse MoE. Attention KV and recurrent state must not be collapsed into a transformer-only cache formula.
• Qwen thinks by default. This guide pins its native template and disables thinking via enable_thinking=false plus --reasoning off. /think and /nothink are not officially supported soft switches. A 4K non-thinking recipe does not preserve/validate full thinking capability; the publisher advises at least 128K context for that purpose.
• The publisher advertises 262,144 native context with extension options. This recipe configures 4,096 total prompt and output, one active sequence and 512 validation output tokens. Longer context, thinking, concurrent users and another cache policy create different configurations.
• Use Qwen’s non-thinking general-task sampler (temperature 0.7, top-p 0.8, top-k 20, min-p 0, presence penalty 1.5 and repetition penalty 1.0). The model card gives conflicting non-thinking reasoning presets in two sections; this recipe avoids that task/mode and uses the consistent general-task preset.
• Operating system packages and build tool dependencies resolve when installed. Save tool versions/build logs with any actual execution. Cloud hardware is AWS r7i.xlarge with four vCPUs and 32 GiB host RAM; four inference threads are allocated. No exact-instance benchmark is claimed.
• Loopback and a service key do not isolate the service from privileged local users. A shared/public service needs separately designed TLS, authentication and operations.
• Single-image description and the inherited supplied-passage text check are smoke checks only, not measured visual/OCR/grounding quality. The server may warm up larger vision buffers than the request; file bytes and the image-token ceiling do not prove fit. Actual prompt/image tokens, encoder memory, speed and quality remain unknown.
• No image preprocessing is delegated to Transformers: libmtmd uses its pinned dynamic-resolution QWEN3VL path, bicubic resize, model spatial merge metadata and CLI image token bounds. Do not transplant Hugging Face min_pixels/max_pixels request keys; this recipe sets server flags.

Download or checksum failure

Verify exact uppercase filename and pinned URL; remove only the incomplete dedicated model file and retry. Do not run a mismatching file.

Architecture unsupported or build fails

Confirm exact pinned source commit, Ubuntu compiler/CMake dependencies and GGUF. Do not substitute an older packaged llama.cpp silently.

Out of memory or context overflow

Stop server, free/provision RAM or reduce context and restart; shorten input/output within configured total. Allocation is not a measured guarantee; record any changed configuration.

Connection refused or HTTP 401

Check foreground server logs, 8080 binding, matching private key and Bearer header. Cloud client must have working SSH tunnel to the correct VM; do not expose the port to repair a tunnel.

Unexpected thinking, template or projector failure

Check exact model/template/projector hashes, native Jinja enable_thinking=false, --reasoning off and --no-mmproj-offload. Confirm image_url content array and data URI. Stop on embedding/architecture mismatch; do not substitute another size projector.

Stop the model

1. 1. Stop the service

   Press Ctrl+C in server terminal. Unset key in each shell. Close the SSH tunnel with Ctrl+C. Stopping the process alone does not stop EC2 billing.

   CommandCopy command

   unset QWEN35_9B_SERVICE_KEY

   Expected result: Server stops and localhost:8080 stops serving.

Uninstall

1. 1. Remove dedicated model, build and credential files

   Back up wanted logs/files first. Remove only this walkthrough directory after confirming its path. Shared apt packages can remain; do not remove other workloads.

   CommandCopy command

   rm -r -- "$HOME/qwen35-9b-vision-hosting"

   Expected result: Dedicated files and local service credential removed; shared system dependencies remain.

Shut down cloud resources

1. 1. End cloud billing resources

   In EC2 us-east-1, stop for later reuse or terminate this exact VM. Review Delete on termination for its root volume. Delete unwanted detached EBS volumes, snapshots and reserved/Elastic/public IP resources. Stopped VM storage and retained resources may still bill. Check Bills/Cost Explorer later for final usage.

   Expected result: Exact instance is stopped/terminated as intended; no unwanted attached/retained billed resources remain. Instance compute stops only after its lifecycle transition, not when llama-server stops.

Related guides

• https://opentools.ai/llms/qwen35-9b

Exact model and requirements

Artifact:

Unsloth (GGUF quantization); Qwen (base model) / Qwen3.5-9B Q4_K_M, Unsloth derivative · checkpoint unsloth/Qwen3.5-9B-GGUF

Publisher revision:

3885219b6810b007914f3a7950a8d1b469d598a5

Artifact digest:

sha256:03b74727a860a56338e042c4420bb3f04b2fec5734175f4cb9fa853daf52b7e8

Download size:

5.29 GiB

Memory requirement (unknown):

Unknown — No exact process peak for this Qwen3.5 GGUF, recurrent-state and text workload.

System RAM / disk:

Unknown / Unknown

Placement:

single device. One CPU host; no tensor/pipeline sharding. CPU-only execution; --device none and zero GPU layers. No disk or accelerator offload. Vision projector also explicitly CPU-only via --no-mmproj-offload.

KV cache:

F16 attention K/V cache; Gated DeltaNet recurrent state is separately runtime-managed, with no claimed state precision or full-cache byte estimate.

Workload:

8,192 token context; Unknown; Unknown; concurrency 1; batch 1; cache unknown; repetitions Unknown.

Tokenizer:

Qwen3.5-9B GGUF tokenizer with separately pinned Qwen native Jinja template · 3885219b6810b007914f3a7950a8d1b469d598a5; template c202236235762e1c871ad0ccb60c8ee5ba337b9a

Sampling / reasoning:

Qwen non-thinking general-task preset: temperature 0.7, top_p 0.8, top_k 20, min_p 0.0, presence_penalty 1.5, repeat_penalty 1.0. Validation max_tokens 512. Non-thinking native Qwen template; temporal video via input_video.data, CPU language/projector and FFmpeg decoded frames. No audio transcript, tools, video generation or MTP.

Runtime prerequisites:

Ubuntu 24.04 x86_64; See prerequisites above.

Official artifact · License

Qwen publishes the base under Apache License 2.0; Unsloth’s derivative metadata declares Apache-2.0 and links that license. Read the full license and retain required license, copyright and attribution notices; mark modifications when redistributing covered work. Unsloth’s repository has no separate LICENSE file at this revision (404); attribution is based on its explicit metadata plus the linked Qwen license. Access is public/ungated as checked; no provider API subscription is required.

• qwen35-chat-template.jinja · 0 GiB · revision c202236235762e1c871ad0ccb60c8ee5ba337b9a
• mmproj-F16.gguf · 0.86 GiB · revision 3885219b6810b007914f3a7950a8d1b469d598a5

• Gated DeltaNet recurrent state, full-attention cache, weights, buffers and OS share CPU system RAM. GGUF file bytes alone do not prove fit.
• F16 cache flags describe attention K/V only; recurrent-state precision/overhead is not measured or inferred. No sibling or GPU throughput is transferred.
• Additional 918166080-byte F16 projector, decoded pixels, vision encoder, image embeddings and warmup buffers require host RAM beyond text-only allocations. No fit, process peak or vision-token measurement exists. 32 GiB host allocation is a walkthrough choice, not a validated minimum.
• Video decoding RGB buffers, multiple visual embeddings, projector and temporal/recurrent state add memory beyond still-image/model files. Full process peak remains unknown.

Install the runtime

1. 1. Build the pinned CPU server

   Install build tools/Python, build FFmpeg n8.0.1 from its exact source commit into this walkthrough directory, then compile the exact llama.cpp CPU server with video and subprocess support. FFmpeg automatic external-library detection and network input are disabled for this local base64-file workflow; built-in H.264 decoding remains enabled. No new packages are installed during this research; these are reader commands. Preserve version/build logs.

   CommandCopy command

   sudo apt-get update
   sudo apt-get install -y build-essential cmake git curl ca-certificates openssl pkg-config python3
   mkdir -p "$HOME/qwen35-9b-video-hosting/models"
   cd "$HOME/qwen35-9b-video-hosting"
   git clone https://github.com/FFmpeg/FFmpeg.git ffmpeg-source
   git -C ffmpeg-source checkout --detach 894da5ca7d742e4429ffb2af534fcda0103ef593
   cd ffmpeg-source
   ./configure --prefix="$HOME/qwen35-9b-video-hosting/ffmpeg-bin" --disable-x86asm --disable-doc --disable-debug --disable-autodetect --disable-network
   make -j 4
   make install
   cd ..
   export PATH="$HOME/qwen35-9b-video-hosting/ffmpeg-bin/bin:$PATH"
   ffmpeg -version
   ffprobe -version
   git clone https://github.com/ggml-org/llama.cpp.git
   cd llama.cpp
   git checkout --detach 620d2225cc436ab0b001e3200e590e3b286ef9ab
   cmake -S . -B build -DCMAKE_BUILD_TYPE=Release -DGGML_METAL=OFF -DGGML_NATIVE=ON -DMTMD_VIDEO=ON -DLLAMA_SUBPROCESS=ON
   cmake --build build --config Release -j 4
   ./build/bin/llama-server --version

   Expected result: Pinned ffmpeg and ffprobe binaries report n8.0.1/source revision; llama-server build succeeds with MTMD_VIDEO and LLAMA_SUBPROCESS enabled. No model inference has run yet.

Download the model

1. 1. Download language weights, native template, matching projector and tiny image fixture

   Verify both public immutable inputs. Qwen base/template and Unsloth quantization have separate revisions and authors. Use the explicit native Jinja template so non-thinking control does not depend on an uninspected embedded-template version. Stop on either hash mismatch. The additional projector is Unsloth-published for this exact 9B derivative repository. The official runtime test fixture is for a private reproducibility check; no redistribution rights to the newspaper photograph are asserted.

   CommandCopy command

   cd "$HOME/qwen35-9b-video-hosting"
   curl --fail --location --output models/Qwen3.5-9B-Q4_K_M.gguf 'https://huggingface.co/unsloth/Qwen3.5-9B-GGUF/resolve/3885219b6810b007914f3a7950a8d1b469d598a5/Qwen3.5-9B-Q4_K_M.gguf'
   curl --fail --location --output models/qwen35-chat-template.jinja 'https://huggingface.co/Qwen/Qwen3.5-9B/raw/c202236235762e1c871ad0ccb60c8ee5ba337b9a/chat_template.jinja'
   sha256sum --check <<'SHA256'
   03b74727a860a56338e042c4420bb3f04b2fec5734175f4cb9fa853daf52b7e8  models/Qwen3.5-9B-Q4_K_M.gguf
   a4aee8afcf2e0711942cf848899be66016f8d14a889ff9ede07bca099c28f715  models/qwen35-chat-template.jinja
   SHA256
   curl --fail --location --output models/mmproj-F16.gguf 'https://huggingface.co/unsloth/Qwen3.5-9B-GGUF/resolve/3885219b6810b007914f3a7950a8d1b469d598a5/mmproj-F16.gguf'
   curl --fail --location --output fixture.jpg 'https://raw.githubusercontent.com/ggml-org/llama.cpp/620d2225cc436ab0b001e3200e590e3b286ef9ab/tools/mtmd/test-1.jpeg'
   sha256sum --check <<'SHA256'
   f70dc3509053962b0d0d3ee8a7eacebf5d60aa560cad78254ae8698516ae029f  models/mmproj-F16.gguf
   2dff664c0c8aaea18aff8cbe7e868845b775e90cdd7a0bac98df709b131deaa3  fixture.jpg
   SHA256

   Expected result: All four immutable files report OK. Projector 918166080 bytes; fixture 124071 bytes. Stop on mismatch.

Configure

1. 1. Create a private service credential

   Generate this only on your own host when following the guide. It is a local service password, not an OpenAI/provider credential. Keep it private and never paste it into tickets. Create it only when absent; preserve an existing key so a repeated setup does not silently rotate access. It is not created during this documentation run.

   CommandCopy command

   cd "$HOME/qwen35-9b-video-hosting"
   umask 077
   if [ ! -e .api-key ]; then
     openssl rand -hex 32 > .api-key
   fi
   test -s .api-key || { echo 'Existing .api-key is empty; restore it before continuing.'; exit 1; }
   chmod 600 .api-key

   Expected result: A non-empty private .api-key file exists; existing key preserved. No secret is printed.

Start the model

1. 1. Start one CPU-only sequence

   Start the exact language model and projector on CPU with non-thinking native template. Keep one sequence, 8,192 total context and 512 output cap. Sample video at 0.5 fps; timestamps are resampled-frame positions emitted by the runtime, not promises of exact original-frame time. Use nominal image min/max 1024 tokens and encoding batch 1,024. A conservative planning allowance is 5 × 1,024 + 512 = 5,632, leaving 2,560 for text/template/timestamps; actual model tokens, rounding and temporal merging are unknown. Preflight the full input before every request; these flags do not stop an oversized clip early.

   CommandCopy command

   cd "$HOME/qwen35-9b-video-hosting"
   export QWEN35_9B_VIDEO_SERVICE_KEY="$(cat .api-key)"
   ./llama.cpp/build/bin/llama-server -m models/Qwen3.5-9B-Q4_K_M.gguf --mmproj models/mmproj-F16.gguf --no-mmproj-offload --image-min-tokens 1024 --image-max-tokens 1024 --alias qwen35-9b --host 127.0.0.1 --port 8080 --api-key "$QWEN35_9B_VIDEO_SERVICE_KEY" --device none -ngl 0 -t 4 -c 8192 -np 1 --cache-type-k f16 --cache-type-v f16 --jinja --chat-template-file models/qwen35-chat-template.jinja --chat-template-kwargs '{"enable_thinking":false}' --reasoning off --reasoning-format none --skip-chat-parsing --temp 0.7 --top-p 0.8 --top-k 20 --min-p 0.0 --presence-penalty 1.5 --repeat-penalty 1.0 --video-fps 0.5 --video-timestamp-interval 2000 --video-ffmpeg-dir "$HOME/qwen35-9b-video-hosting/ffmpeg-bin/bin" --mmproj-device none --mtmd-batch-max-tokens 1024

   Expected result: Model initializes and listener starts. Pass health and authenticated chat checks before relying on the service. Command-line credentials may be visible to privileged/local users; trusted host required.

Verify the setup

1. 1. Check readiness and authenticated text generation

   Use a second server terminal and the private service key. Send only plain text. Check final response against the supplied source passage and verify finish_reason is not length; no exact wording is guaranteed. Do not expect reasoning_content in raw mode. If extended thinking appears, non-thinking setup validation failed.

   CommandCopy command

   cd "$HOME/qwen35-9b-video-hosting"
   export QWEN35_9B_VIDEO_SERVICE_KEY="$(cat .api-key)"
   curl --fail http://127.0.0.1:8080/health
   curl --fail http://127.0.0.1:8080/v1/chat/completions -H "Authorization: Bearer $QWEN35_9B_VIDEO_SERVICE_KEY" -H 'Content-Type: application/json' -d '{"model":"qwen35-9b","messages":[{"role":"user","content":"Summarize this supplied passage in one sentence without adding facts: The community library opens at 9am on weekdays and closes at 6pm. On Saturday it opens at 10am and closes at 4pm. It is closed on Sunday."}],"max_tokens":512,"temperature":0.7,"top_k":20,"top_p":0.8,"min_p":0.0,"presence_penalty":1.5,"repeat_penalty":1.0}'
   curl -o /dev/null -s -w '%{http_code}\n' http://127.0.0.1:8080/v1/chat/completions -H 'Content-Type: application/json' -d '{"model":"qwen35-9b","messages":[{"role":"user","content":"Hello"}]}'

   Expected result: Health HTTP 200 when ready; authenticated response contains non-empty choices[0].message.content with an accurate final summary and finish_reason not length, without a generated extended thinking trace. Unauthenticated chat returns HTTP 401. Verify facts yourself; output can be wrong.

2. 2. Confirm loaded video capability

   Health proves the process is ready; authenticated /props must also report both vision and video capability before video input. Stop if either is absent/false.

   CommandCopy command

   cd "$HOME/qwen35-9b-video-hosting"
   export QWEN35_9B_VIDEO_SERVICE_KEY="$(cat .api-key)"
   curl --fail http://127.0.0.1:8080/props -H "Authorization: Bearer $QWEN35_9B_VIDEO_SERVICE_KEY" --output loaded-props.json
   python3 - <<'PYPROPS'
   import json,pathlib
   p=json.loads(pathlib.Path('loaded-props.json').read_text())
   assert p['modalities']['vision'] is True and p['modalities']['video'] is True
   print('Loaded vision and video capabilities confirmed')
   PYPROPS

   Expected result: Both modalities.vision and modalities.video are true. This is a required reader check, not an observed result from this authoring run.

3. 3. Verify all sampled frames before video submission

   Download and hash-check the whole immutable moving MP4. ffprobe verifies source metadata. Decode the entire file through the same fixed fps=0.500000 RGB24 filter as the native helper; count all bytes, with no -frames or artificial stop. Reject a changed source, oversized duration, partial frame or any count other than five. This is media decoding only, not model inference.

   CommandCopy command

   cd "$HOME/qwen35-9b-video-hosting"
   export PATH="$HOME/qwen35-9b-video-hosting/ffmpeg-bin/bin:$PATH"
   curl --fail --location --output fixture.mp4 'https://raw.githubusercontent.com/ggml-org/llama.cpp/620d2225cc436ab0b001e3200e590e3b286ef9ab/tools/mtmd/test-3.mp4'
   printf '%s  %s\n' '95a9776a0e7f8d6ab1ba67a31dbb09a4c04ffda3c853fd933624e3496fa970a5' 'fixture.mp4' | sha256sum --check
   ffprobe -v error -select_streams v:0 -show_entries stream=width,height,codec_name:format=duration -of json fixture.mp4 > fixture-metadata.json
   ffmpeg -nostdin -y -v error -i fixture.mp4 -vf fps=0.500000 -an -f rawvideo -pix_fmt rgb24 fixture-rgb.raw
   python3 - <<'PYPREFLIGHT'
   import json,pathlib
   m=json.loads(pathlib.Path('fixture-metadata.json').read_text())
   s=m['streams'][0]
   assert s['width']==854 and s['height']==358 and s['codec_name']=='h264'
   assert 10.02 <= float(m['format']['duration']) <= 10.04 and float(m['format']['duration']) <= 10.1
   size=pathlib.Path('fixture-rgb.raw').stat().st_size
   frame_bytes=854*358*3
   assert size % frame_bytes == 0, 'partial decoded frame'
   assert size // frame_bytes == 5, 'reject: expected exactly five sampled frames'
   print('Preflight OK: complete five-frame bounded input')
   PYPREFLIGHT

   Expected result: Fixture SHA OK; metadata matches; preflight reports exactly five complete sampled frames. If any assertion fails, stop before video submission. No generated frame limit hides additional decoded frames.

4. 4. Check temporal ordering with one short video

   Create input_video.data as raw MP4 base64 using the native API schema and send it only to this private local endpoint. The author visually inspected source frames, not a model output. Ask for observable scene order; do not require character names, dialogue or inferred motives. FFmpeg sampling can choose source frames near 1, 3, 5, 7 and 9 seconds; runtime timestamps are resampled positions and may differ from original frame times.

   CommandCopy command

   cd "$HOME/qwen35-9b-video-hosting"
   python3 - <<'PYREQUEST'
   import base64,json,pathlib
   video=base64.b64encode(pathlib.Path('fixture.mp4').read_bytes()).decode()
   request={'model':'qwen35-9b','messages':[{'role':'user','content':[{'type':'text','text':'Describe the beginning, middle and end of this short clip. Explain observable changes in the man’s position and his interaction with the chair. Do not infer dialogue, names or motives.'},{'type':'input_video','input_video':{'data':video}}]}],'max_tokens':512,'temperature':0.7,'top_p':0.8,'top_k':20,'min_p':0.0,'presence_penalty':1.5,'repeat_penalty':1.0}
   pathlib.Path('video-request.json').write_text(json.dumps(request))
   PYREQUEST
   export QWEN35_9B_VIDEO_SERVICE_KEY="$(cat .api-key)"
   curl --fail http://127.0.0.1:8080/v1/chat/completions -H "Authorization: Bearer $QWEN35_9B_VIDEO_SERVICE_KEY" -H 'Content-Type: application/json' --data-binary @video-request.json

   Expected result: Completed message.content correctly orders initially seated/caped man beside large gray character, later hand reaching/pulling chair-side handle, then man out of chair and standing/moving around the room. Inspect response against the clip/frames; no exact phrasing or exact source timestamps required. Wrong order, missing temporal change, context error or finish_reason length fails this narrow functional check. No model result is claimed here.

Performance evidence

No compatible measurement available. Documented support and estimated memory do not establish response speed.

Limits and troubleshooting

• Documented configuration only. No weights, build, inference, full-fit, throughput or latency benchmark was executed by OpenTools.
• Unsloth Q4_K_M GGUF is 5,680,522,464 bytes. It is a derivative of Qwen/Qwen3.5-9B with separate quantizer revision; no equivalence to full precision quality is claimed.
• The native 9B model is a dense language backbone with a vision encoder, 32 layers and alternating three Gated DeltaNet layers per full-attention layer. Family-wide MoE marketing does not make this 9B checkpoint a sparse MoE. Attention KV and recurrent state must not be collapsed into a transformer-only cache formula.
• Qwen thinks by default. This guide pins its native template and disables thinking via enable_thinking=false plus --reasoning off. /think and /nothink are not officially supported soft switches. This 8K non-thinking recipe does not preserve/validate full thinking capability; the publisher advises at least 128K context for that purpose.
• Use Qwen’s non-thinking general-task sampler (temperature 0.7, top-p 0.8, top-k 20, min-p 0, presence penalty 1.5 and repetition penalty 1.0). The model card gives conflicting non-thinking reasoning presets in two sections; this recipe avoids that task/mode and uses the consistent general-task preset.
• Operating system packages and build tool dependencies resolve when installed. Save tool versions/build logs with any actual execution. Local hardware is Intel Core i5-12400 with four inference threads allocated; no exact-hardware benchmark is claimed.
• Loopback and a service key do not isolate the service from privileged local users. A shared/public service needs separately designed TLS, authentication and operations.
• The temporal-video ordering check and supplied-passage text check are smoke checks only, not measured visual/OCR/grounding quality. The server may warm up larger vision buffers than the request; file bytes and the image-token ceiling do not prove fit. Actual prompt/image tokens, encoder memory, speed and quality remain unknown.
• No image preprocessing is delegated to Transformers: libmtmd uses its pinned dynamic-resolution QWEN3VL path, bicubic resize, model spatial merge metadata and CLI image token bounds. Do not transplant Hugging Face min_pixels/max_pixels request keys; this recipe sets server flags.
• This is a distinct temporal-video job: a real moving clip and ordered events, not a still image wrapped as video. It does not add audio understanding, video generation, arbitrary long video or validated multi-clip analysis.
• Source fixture is 10.03 seconds. Runtime decodes the whole input and has no hard request duration/frame limit, so preflight duration and all sampled frames are mandatory. 0.5 fps can miss fast events; the guide checks only this coarse seated-to-handle-to-standing sequence.
• The nominal five-frame image budget at 1,024 each plus 512 output fits within the 8K planning allocation with headroom, but actual image tokens, merge behavior, template and timestamp overhead are unmeasured. Runtime can temporally merge compatible QWEN3VL frames; no token discount or fit proof is inferred.
• Use non-thinking native Qwen template and general-task sampler. 8K does not exercise the publisher’s full advertised context or 128K thinking recommendation. No image/dense-sibling/video throughput or memory result is transferred.
• The official repository distributes the fixture, but media rights are unknown. This guide links it for private validation and does not authorize public rehosting. Source-frame visual inspection is human evidence, not executed inference.

Download or checksum failure

Verify exact uppercase filename and pinned URL; remove only the incomplete dedicated model file and retry. Do not run a mismatching file.

Architecture unsupported or build fails

Confirm exact pinned source commit, Ubuntu compiler/CMake dependencies and GGUF. Do not substitute an older packaged llama.cpp silently.

Out of memory or context overflow

Stop server, free/provision RAM or reduce context and restart; shorten input/output within configured total. Allocation is not a measured guarantee; record any changed configuration.

Connection refused or HTTP 401

Check foreground server logs, 8080 binding, matching private key and Bearer header. Run the client on this same desktop and keep the loopback listener.

Unexpected thinking, template or projector failure

Check exact model/template/projector hashes, native Jinja enable_thinking=false, --reasoning off and --no-mmproj-offload. Confirm native input_video.data contains the base64 MP4 and the full five-frame preflight passed. Stop on embedding/architecture mismatch; do not substitute another size projector.

Video not supported, FFmpeg error, too many frames or wrong event order

Confirm video/subprocess build options, pinned ffmpeg and ffprobe in --video-ffmpeg-dir, exact MP4 SHA and full five-frame preflight. Use native input_video.data, not image_url or a plain path. Never ignore a failed duration/count check or cap decoding to hide extra frames. Wrong/truncated ordering fails validation; inspect sampled-frame limitations before changing fps/context, which changes this configuration.

Stop the model

1. 1. Stop the service

   Press Ctrl+C in server terminal. Unset key in each shell. Confirm this desktop localhost:8080 no longer serves the model.

   CommandCopy command

   unset QWEN35_9B_VIDEO_SERVICE_KEY

   Expected result: Server stops and localhost:8080 stops serving.

Uninstall

1. 1. Remove dedicated model, build and credential files

   Back up wanted logs/files first. Remove only this walkthrough directory after confirming its path. Shared apt packages can remain; do not remove other workloads.

   CommandCopy command

   rm -r -- "$HOME/qwen35-9b-video-hosting"

   Expected result: Dedicated files and local service credential removed; shared system dependencies remain.

Related guides

• https://opentools.ai/llms/qwen35-9b

Exact model and requirements

Artifact:

Unsloth (GGUF quantization); Qwen (base model) / Qwen3.5-9B Q4_K_M, Unsloth derivative · checkpoint unsloth/Qwen3.5-9B-GGUF

Publisher revision:

3885219b6810b007914f3a7950a8d1b469d598a5

Artifact digest:

sha256:03b74727a860a56338e042c4420bb3f04b2fec5734175f4cb9fa853daf52b7e8

Download size:

5.29 GiB

Memory requirement (unknown):

Unknown — No exact process peak for this Qwen3.5 GGUF, recurrent-state and text workload.

System RAM / disk:

Unknown / Unknown

Placement:

single device. One CPU host; no tensor/pipeline sharding. CPU-only execution; --device none and zero GPU layers. No disk or accelerator offload. Vision projector also explicitly CPU-only via --no-mmproj-offload.

KV cache:

F16 attention K/V cache; Gated DeltaNet recurrent state is separately runtime-managed, with no claimed state precision or full-cache byte estimate.

Workload:

8,192 token context; Unknown; Unknown; concurrency 1; batch 1; cache unknown; repetitions Unknown.

Tokenizer:

Qwen3.5-9B GGUF tokenizer with separately pinned Qwen native Jinja template · 3885219b6810b007914f3a7950a8d1b469d598a5; template c202236235762e1c871ad0ccb60c8ee5ba337b9a

Sampling / reasoning:

Qwen non-thinking general-task preset: temperature 0.7, top_p 0.8, top_k 20, min_p 0.0, presence_penalty 1.5, repeat_penalty 1.0. Validation max_tokens 512. Non-thinking native Qwen template; temporal video via input_video.data, CPU language/projector and FFmpeg decoded frames. No audio transcript, tools, video generation or MTP.

Runtime prerequisites:

Ubuntu 24.04 x86_64; See prerequisites above.

Official artifact · License

Qwen publishes the base under Apache License 2.0; Unsloth’s derivative metadata declares Apache-2.0 and links that license. Read the full license and retain required license, copyright and attribution notices; mark modifications when redistributing covered work. Unsloth’s repository has no separate LICENSE file at this revision (404); attribution is based on its explicit metadata plus the linked Qwen license. Access is public/ungated as checked; no provider API subscription is required.

• qwen35-chat-template.jinja · 0 GiB · revision c202236235762e1c871ad0ccb60c8ee5ba337b9a
• mmproj-F16.gguf · 0.86 GiB · revision 3885219b6810b007914f3a7950a8d1b469d598a5

• Gated DeltaNet recurrent state, full-attention cache, weights, buffers and OS share CPU system RAM. GGUF file bytes alone do not prove fit.
• F16 cache flags describe attention K/V only; recurrent-state precision/overhead is not measured or inferred. No sibling or GPU throughput is transferred.
• Additional 918166080-byte F16 projector, decoded pixels, vision encoder, image embeddings and warmup buffers require host RAM beyond text-only allocations. No fit, process peak or vision-token measurement exists. 32 GiB host allocation is a walkthrough choice, not a validated minimum.
• Video decoding RGB buffers, multiple visual embeddings, projector and temporal/recurrent state add memory beyond still-image/model files. Full process peak remains unknown.

Install the runtime

1. 1. Build the pinned CPU server

   Install build tools/Python, build FFmpeg n8.0.1 from its exact source commit into this walkthrough directory, then compile the exact llama.cpp CPU server with video and subprocess support. FFmpeg automatic external-library detection and network input are disabled for this local base64-file workflow; built-in H.264 decoding remains enabled. No new packages are installed during this research; these are reader commands. Preserve version/build logs.

   CommandCopy command

   sudo apt-get update
   sudo apt-get install -y build-essential cmake git curl ca-certificates openssl pkg-config python3
   mkdir -p "$HOME/qwen35-9b-video-hosting/models"
   cd "$HOME/qwen35-9b-video-hosting"
   git clone https://github.com/FFmpeg/FFmpeg.git ffmpeg-source
   git -C ffmpeg-source checkout --detach 894da5ca7d742e4429ffb2af534fcda0103ef593
   cd ffmpeg-source
   ./configure --prefix="$HOME/qwen35-9b-video-hosting/ffmpeg-bin" --disable-x86asm --disable-doc --disable-debug --disable-autodetect --disable-network
   make -j 4
   make install
   cd ..
   export PATH="$HOME/qwen35-9b-video-hosting/ffmpeg-bin/bin:$PATH"
   ffmpeg -version
   ffprobe -version
   git clone https://github.com/ggml-org/llama.cpp.git
   cd llama.cpp
   git checkout --detach 620d2225cc436ab0b001e3200e590e3b286ef9ab
   cmake -S . -B build -DCMAKE_BUILD_TYPE=Release -DGGML_METAL=OFF -DGGML_NATIVE=ON -DMTMD_VIDEO=ON -DLLAMA_SUBPROCESS=ON
   cmake --build build --config Release -j 4
   ./build/bin/llama-server --version

   Expected result: Pinned ffmpeg and ffprobe binaries report n8.0.1/source revision; llama-server build succeeds with MTMD_VIDEO and LLAMA_SUBPROCESS enabled. No model inference has run yet.

Download the model

1. 1. Download language weights, native template, matching projector and tiny image fixture

   Verify both public immutable inputs. Qwen base/template and Unsloth quantization have separate revisions and authors. Use the explicit native Jinja template so non-thinking control does not depend on an uninspected embedded-template version. Stop on either hash mismatch. The additional projector is Unsloth-published for this exact 9B derivative repository. The official runtime test fixture is for a private reproducibility check; no redistribution rights to the newspaper photograph are asserted.

   CommandCopy command

   cd "$HOME/qwen35-9b-video-hosting"
   curl --fail --location --output models/Qwen3.5-9B-Q4_K_M.gguf 'https://huggingface.co/unsloth/Qwen3.5-9B-GGUF/resolve/3885219b6810b007914f3a7950a8d1b469d598a5/Qwen3.5-9B-Q4_K_M.gguf'
   curl --fail --location --output models/qwen35-chat-template.jinja 'https://huggingface.co/Qwen/Qwen3.5-9B/raw/c202236235762e1c871ad0ccb60c8ee5ba337b9a/chat_template.jinja'
   sha256sum --check <<'SHA256'
   03b74727a860a56338e042c4420bb3f04b2fec5734175f4cb9fa853daf52b7e8  models/Qwen3.5-9B-Q4_K_M.gguf
   a4aee8afcf2e0711942cf848899be66016f8d14a889ff9ede07bca099c28f715  models/qwen35-chat-template.jinja
   SHA256
   curl --fail --location --output models/mmproj-F16.gguf 'https://huggingface.co/unsloth/Qwen3.5-9B-GGUF/resolve/3885219b6810b007914f3a7950a8d1b469d598a5/mmproj-F16.gguf'
   curl --fail --location --output fixture.jpg 'https://raw.githubusercontent.com/ggml-org/llama.cpp/620d2225cc436ab0b001e3200e590e3b286ef9ab/tools/mtmd/test-1.jpeg'
   sha256sum --check <<'SHA256'
   f70dc3509053962b0d0d3ee8a7eacebf5d60aa560cad78254ae8698516ae029f  models/mmproj-F16.gguf
   2dff664c0c8aaea18aff8cbe7e868845b775e90cdd7a0bac98df709b131deaa3  fixture.jpg
   SHA256

   Expected result: All four immutable files report OK. Projector 918166080 bytes; fixture 124071 bytes. Stop on mismatch.

Configure

1. 1. Create a private service credential

   Generate this only on your own host when following the guide. It is a local service password, not an OpenAI/provider credential. Keep it private and never paste it into tickets. Create it only when absent; preserve an existing key so a repeated setup does not silently rotate access. It is not created during this documentation run.

   CommandCopy command

   cd "$HOME/qwen35-9b-video-hosting"
   umask 077
   if [ ! -e .api-key ]; then
     openssl rand -hex 32 > .api-key
   fi
   test -s .api-key || { echo 'Existing .api-key is empty; restore it before continuing.'; exit 1; }
   chmod 600 .api-key

   Expected result: A non-empty private .api-key file exists; existing key preserved. No secret is printed.

Start the model

1. 1. Start one CPU-only sequence

   Start the exact language model and projector on CPU with non-thinking native template. Keep one sequence, 8,192 total context and 512 output cap. Sample video at 0.5 fps; timestamps are resampled-frame positions emitted by the runtime, not promises of exact original-frame time. Use nominal image min/max 1024 tokens and encoding batch 1,024. A conservative planning allowance is 5 × 1,024 + 512 = 5,632, leaving 2,560 for text/template/timestamps; actual model tokens, rounding and temporal merging are unknown. Preflight the full input before every request; these flags do not stop an oversized clip early.

   CommandCopy command

   cd "$HOME/qwen35-9b-video-hosting"
   export QWEN35_9B_VIDEO_SERVICE_KEY="$(cat .api-key)"
   ./llama.cpp/build/bin/llama-server -m models/Qwen3.5-9B-Q4_K_M.gguf --mmproj models/mmproj-F16.gguf --no-mmproj-offload --image-min-tokens 1024 --image-max-tokens 1024 --alias qwen35-9b --host 127.0.0.1 --port 8080 --api-key "$QWEN35_9B_VIDEO_SERVICE_KEY" --device none -ngl 0 -t 4 -c 8192 -np 1 --cache-type-k f16 --cache-type-v f16 --jinja --chat-template-file models/qwen35-chat-template.jinja --chat-template-kwargs '{"enable_thinking":false}' --reasoning off --reasoning-format none --skip-chat-parsing --temp 0.7 --top-p 0.8 --top-k 20 --min-p 0.0 --presence-penalty 1.5 --repeat-penalty 1.0 --video-fps 0.5 --video-timestamp-interval 2000 --video-ffmpeg-dir "$HOME/qwen35-9b-video-hosting/ffmpeg-bin/bin" --mmproj-device none --mtmd-batch-max-tokens 1024

   Expected result: Model initializes and listener starts. Pass health and authenticated chat checks before relying on the service. Command-line credentials may be visible to privileged/local users; trusted host required.

Verify the setup

1. 1. Check readiness and authenticated text generation

   Use a second server terminal and the private service key. Send only plain text. Check final response against the supplied source passage and verify finish_reason is not length; no exact wording is guaranteed. Do not expect reasoning_content in raw mode. If extended thinking appears, non-thinking setup validation failed.

   CommandCopy command

   cd "$HOME/qwen35-9b-video-hosting"
   export QWEN35_9B_VIDEO_SERVICE_KEY="$(cat .api-key)"
   curl --fail http://127.0.0.1:8080/health
   curl --fail http://127.0.0.1:8080/v1/chat/completions -H "Authorization: Bearer $QWEN35_9B_VIDEO_SERVICE_KEY" -H 'Content-Type: application/json' -d '{"model":"qwen35-9b","messages":[{"role":"user","content":"Summarize this supplied passage in one sentence without adding facts: The community library opens at 9am on weekdays and closes at 6pm. On Saturday it opens at 10am and closes at 4pm. It is closed on Sunday."}],"max_tokens":512,"temperature":0.7,"top_k":20,"top_p":0.8,"min_p":0.0,"presence_penalty":1.5,"repeat_penalty":1.0}'
   curl -o /dev/null -s -w '%{http_code}\n' http://127.0.0.1:8080/v1/chat/completions -H 'Content-Type: application/json' -d '{"model":"qwen35-9b","messages":[{"role":"user","content":"Hello"}]}'

   Expected result: Health HTTP 200 when ready; authenticated response contains non-empty choices[0].message.content with an accurate final summary and finish_reason not length, without a generated extended thinking trace. Unauthenticated chat returns HTTP 401. Verify facts yourself; output can be wrong.

2. 2. Confirm loaded video capability

   Health proves the process is ready; authenticated /props must also report both vision and video capability before video input. Stop if either is absent/false.

   CommandCopy command

   cd "$HOME/qwen35-9b-video-hosting"
   export QWEN35_9B_VIDEO_SERVICE_KEY="$(cat .api-key)"
   curl --fail http://127.0.0.1:8080/props -H "Authorization: Bearer $QWEN35_9B_VIDEO_SERVICE_KEY" --output loaded-props.json
   python3 - <<'PYPROPS'
   import json,pathlib
   p=json.loads(pathlib.Path('loaded-props.json').read_text())
   assert p['modalities']['vision'] is True and p['modalities']['video'] is True
   print('Loaded vision and video capabilities confirmed')
   PYPROPS

   Expected result: Both modalities.vision and modalities.video are true. This is a required reader check, not an observed result from this authoring run.

3. 3. Verify all sampled frames before video submission

   Download and hash-check the whole immutable moving MP4. ffprobe verifies source metadata. Decode the entire file through the same fixed fps=0.500000 RGB24 filter as the native helper; count all bytes, with no -frames or artificial stop. Reject a changed source, oversized duration, partial frame or any count other than five. This is media decoding only, not model inference.

   CommandCopy command

   cd "$HOME/qwen35-9b-video-hosting"
   export PATH="$HOME/qwen35-9b-video-hosting/ffmpeg-bin/bin:$PATH"
   curl --fail --location --output fixture.mp4 'https://raw.githubusercontent.com/ggml-org/llama.cpp/620d2225cc436ab0b001e3200e590e3b286ef9ab/tools/mtmd/test-3.mp4'
   printf '%s  %s\n' '95a9776a0e7f8d6ab1ba67a31dbb09a4c04ffda3c853fd933624e3496fa970a5' 'fixture.mp4' | sha256sum --check
   ffprobe -v error -select_streams v:0 -show_entries stream=width,height,codec_name:format=duration -of json fixture.mp4 > fixture-metadata.json
   ffmpeg -nostdin -y -v error -i fixture.mp4 -vf fps=0.500000 -an -f rawvideo -pix_fmt rgb24 fixture-rgb.raw
   python3 - <<'PYPREFLIGHT'
   import json,pathlib
   m=json.loads(pathlib.Path('fixture-metadata.json').read_text())
   s=m['streams'][0]
   assert s['width']==854 and s['height']==358 and s['codec_name']=='h264'
   assert 10.02 <= float(m['format']['duration']) <= 10.04 and float(m['format']['duration']) <= 10.1
   size=pathlib.Path('fixture-rgb.raw').stat().st_size
   frame_bytes=854*358*3
   assert size % frame_bytes == 0, 'partial decoded frame'
   assert size // frame_bytes == 5, 'reject: expected exactly five sampled frames'
   print('Preflight OK: complete five-frame bounded input')
   PYPREFLIGHT

   Expected result: Fixture SHA OK; metadata matches; preflight reports exactly five complete sampled frames. If any assertion fails, stop before video submission. No generated frame limit hides additional decoded frames.

4. 4. Check temporal ordering with one short video

   Create input_video.data as raw MP4 base64 using the native API schema and send it only to this private local endpoint. The author visually inspected source frames, not a model output. Ask for observable scene order; do not require character names, dialogue or inferred motives. FFmpeg sampling can choose source frames near 1, 3, 5, 7 and 9 seconds; runtime timestamps are resampled positions and may differ from original frame times.

   CommandCopy command

   cd "$HOME/qwen35-9b-video-hosting"
   python3 - <<'PYREQUEST'
   import base64,json,pathlib
   video=base64.b64encode(pathlib.Path('fixture.mp4').read_bytes()).decode()
   request={'model':'qwen35-9b','messages':[{'role':'user','content':[{'type':'text','text':'Describe the beginning, middle and end of this short clip. Explain observable changes in the man’s position and his interaction with the chair. Do not infer dialogue, names or motives.'},{'type':'input_video','input_video':{'data':video}}]}],'max_tokens':512,'temperature':0.7,'top_p':0.8,'top_k':20,'min_p':0.0,'presence_penalty':1.5,'repeat_penalty':1.0}
   pathlib.Path('video-request.json').write_text(json.dumps(request))
   PYREQUEST
   export QWEN35_9B_VIDEO_SERVICE_KEY="$(cat .api-key)"
   curl --fail http://127.0.0.1:8080/v1/chat/completions -H "Authorization: Bearer $QWEN35_9B_VIDEO_SERVICE_KEY" -H 'Content-Type: application/json' --data-binary @video-request.json

   Expected result: Completed message.content correctly orders initially seated/caped man beside large gray character, later hand reaching/pulling chair-side handle, then man out of chair and standing/moving around the room. Inspect response against the clip/frames; no exact phrasing or exact source timestamps required. Wrong order, missing temporal change, context error or finish_reason length fails this narrow functional check. No model result is claimed here.

Performance evidence

No compatible measurement available. Documented support and estimated memory do not establish response speed.

Limits and troubleshooting

• Documented configuration only. No weights, build, inference, full-fit, throughput or latency benchmark was executed by OpenTools.
• Unsloth Q4_K_M GGUF is 5,680,522,464 bytes. It is a derivative of Qwen/Qwen3.5-9B with separate quantizer revision; no equivalence to full precision quality is claimed.
• The native 9B model is a dense language backbone with a vision encoder, 32 layers and alternating three Gated DeltaNet layers per full-attention layer. Family-wide MoE marketing does not make this 9B checkpoint a sparse MoE. Attention KV and recurrent state must not be collapsed into a transformer-only cache formula.
• Qwen thinks by default. This guide pins its native template and disables thinking via enable_thinking=false plus --reasoning off. /think and /nothink are not officially supported soft switches. This 8K non-thinking recipe does not preserve/validate full thinking capability; the publisher advises at least 128K context for that purpose.
• Use Qwen’s non-thinking general-task sampler (temperature 0.7, top-p 0.8, top-k 20, min-p 0, presence penalty 1.5 and repetition penalty 1.0). The model card gives conflicting non-thinking reasoning presets in two sections; this recipe avoids that task/mode and uses the consistent general-task preset.
• Operating system packages and build tool dependencies resolve when installed. Save tool versions/build logs with any actual execution. Cloud hardware is AWS r7i.xlarge with four vCPUs and 32 GiB host RAM; four inference threads are allocated. No exact-instance benchmark is claimed.
• Loopback and a service key do not isolate the service from privileged local users. A shared/public service needs separately designed TLS, authentication and operations.
• The temporal-video ordering check and supplied-passage text check are smoke checks only, not measured visual/OCR/grounding quality. The server may warm up larger vision buffers than the request; file bytes and the image-token ceiling do not prove fit. Actual prompt/image tokens, encoder memory, speed and quality remain unknown.
• No image preprocessing is delegated to Transformers: libmtmd uses its pinned dynamic-resolution QWEN3VL path, bicubic resize, model spatial merge metadata and CLI image token bounds. Do not transplant Hugging Face min_pixels/max_pixels request keys; this recipe sets server flags.
• This is a distinct temporal-video job: a real moving clip and ordered events, not a still image wrapped as video. It does not add audio understanding, video generation, arbitrary long video or validated multi-clip analysis.
• Source fixture is 10.03 seconds. Runtime decodes the whole input and has no hard request duration/frame limit, so preflight duration and all sampled frames are mandatory. 0.5 fps can miss fast events; the guide checks only this coarse seated-to-handle-to-standing sequence.
• The nominal five-frame image budget at 1,024 each plus 512 output fits within the 8K planning allocation with headroom, but actual image tokens, merge behavior, template and timestamp overhead are unmeasured. Runtime can temporally merge compatible QWEN3VL frames; no token discount or fit proof is inferred.
• Use non-thinking native Qwen template and general-task sampler. 8K does not exercise the publisher’s full advertised context or 128K thinking recommendation. No image/dense-sibling/video throughput or memory result is transferred.
• The official repository distributes the fixture, but media rights are unknown. This guide links it for private validation and does not authorize public rehosting. Source-frame visual inspection is human evidence, not executed inference.

Download or checksum failure

Verify exact uppercase filename and pinned URL; remove only the incomplete dedicated model file and retry. Do not run a mismatching file.

Architecture unsupported or build fails

Confirm exact pinned source commit, Ubuntu compiler/CMake dependencies and GGUF. Do not substitute an older packaged llama.cpp silently.

Out of memory or context overflow

Stop server, free/provision RAM or reduce context and restart; shorten input/output within configured total. Allocation is not a measured guarantee; record any changed configuration.

Connection refused or HTTP 401

Check foreground server logs, 8080 binding, matching private key and Bearer header. Cloud client must have working SSH tunnel to the correct VM; do not expose the port to repair a tunnel.

Unexpected thinking, template or projector failure

Check exact model/template/projector hashes, native Jinja enable_thinking=false, --reasoning off and --no-mmproj-offload. Confirm the content array contains native input_video with raw MP4 base64 in input_video.data. Stop on embedding/architecture mismatch; do not substitute another size projector.

Video not supported, FFmpeg error, too many frames or wrong event order

Confirm video/subprocess build options, pinned ffmpeg and ffprobe in --video-ffmpeg-dir, exact MP4 SHA and full five-frame preflight. Use native input_video.data, not image_url or a plain path. Never ignore a failed duration/count check or cap decoding to hide extra frames. Wrong/truncated ordering fails validation; inspect sampled-frame limitations before changing fps/context, which changes this configuration.

Stop the model

1. 1. Stop the service

   Press Ctrl+C in server terminal. Unset key in each shell. Close the SSH tunnel with Ctrl+C. Stopping the process alone does not stop EC2 billing.

   CommandCopy command

   unset QWEN35_9B_VIDEO_SERVICE_KEY

   Expected result: Server stops and localhost:8080 stops serving.

Uninstall

1. 1. Remove dedicated model, build and credential files

   Back up wanted logs/files first. Remove only this walkthrough directory after confirming its path. Shared apt packages can remain; do not remove other workloads.

   CommandCopy command

   rm -r -- "$HOME/qwen35-9b-video-hosting"

   Expected result: Dedicated files and local service credential removed; shared system dependencies remain.

Shut down cloud resources

1. 1. End cloud billing resources

   In EC2 us-east-1, stop for later reuse or terminate this exact VM. Review Delete on termination for its root volume. Delete unwanted detached EBS volumes, snapshots and reserved/Elastic/public IP resources. Stopped VM storage and retained resources may still bill. Check Bills/Cost Explorer later for final usage.

   Expected result: Exact instance is stopped/terminated as intended; no unwanted attached/retained billed resources remain. Instance compute stops only after its lifecycle transition, not when llama-server stops.

Related guides

• https://opentools.ai/llms/qwen35-9b