Qwen3.5-35B-A3B

Exact model and requirements

Artifact:

Unsloth (GGUF quantization); Qwen (base model) / Qwen3.5-35B-A3B Q4_K_M, Unsloth derivative · checkpoint unsloth/Qwen3.5-35B-A3B-GGUF

Publisher revision:

bc014a17be43adabd7066b7a86075ff935c6a4e2

Artifact digest:

sha256:3b46d1066bc91cc2d613e3bc22ce691dd77e6f0d33c9060690d24ce6de494375

Download size:

20.5 GiB

Memory requirement (unknown):

Unknown — No exact CPU process peak for this complete MoE GGUF and hybrid cache configuration.

System RAM / disk:

Unknown / Unknown

Placement:

single device. One CPU host; no tensor/pipeline sharding. CPU-only execution; --device none and zero GPU layers. No disk or accelerator offload.

KV cache:

F16 attention K/V cache; Gated DeltaNet recurrent state is separately runtime-managed, with no claimed state precision or full-cache byte estimate.

Workload:

4,096 token context; Unknown; Unknown; concurrency 1; batch 1; cache unknown; repetitions Unknown.

Tokenizer:

Qwen3.5-35B-A3B GGUF tokenizer with separately pinned native 35B-A3B Jinja template · bc014a17be43adabd7066b7a86075ff935c6a4e2; template 59d61f3ce65a6d9863b86d2e96597125219dc754

Sampling / reasoning:

Qwen non-thinking general-task preset: temperature 0.7, top_p 0.8, top_k 20, min_p 0.0, presence_penalty 1.5, repeat_penalty 1.0. Validation max_tokens 512. Non-thinking text-only mode, independently supported by this 35B-A3B native template: enable_thinking=false and --reasoning off. No /nothink soft switch, vision projector, tool parser or MTP.

Runtime prerequisites:

Ubuntu 24.04 x86_64; See prerequisites above.

Official artifact · License

Qwen publishes Qwen3.5-35B-A3B under Apache License 2.0; Unsloth’s derivative metadata explicitly links that base and Apache-2.0 license. Read the full license, retain applicable license/copyright/attribution notices and mark modifications when redistributing covered work. Public access is not a substitute for license compliance. No provider inference subscription is required for the public artifact.

• qwen35-chat-template.jinja · 0 GiB · revision 59d61f3ce65a6d9863b86d2e96597125219dc754

• The complete 22,016,023,168-byte model plus attention/recurrent state, buffers, OS and setup files needs resources. Three billion active parameters does not shrink full model storage to a 3B model.
• No transformer-only KV estimate or dense/LFM sibling observation is used. Full working-memory fit, latency and speed remain unknown.

Install the runtime

1. 1. Build the pinned CPU server

   Install build dependencies and compile the exact official source revision. These packages are system shared dependencies; no CUDA/GPU driver is required. GGML_NATIVE=ON selects -march=native for this GNU/Linux x86 build, compiling for the host CPU. Build and run on the same machine; no AVX-512 requirement is imposed on the i5-12400.

   CommandCopy command

   sudo apt-get update
   sudo apt-get install -y build-essential cmake git curl ca-certificates openssl
   mkdir -p "$HOME/qwen35-35ba3b-hosting/models"
   cd "$HOME/qwen35-35ba3b-hosting"
   git clone https://github.com/ggml-org/llama.cpp.git
   cd llama.cpp
   git checkout --detach 620d2225cc436ab0b001e3200e590e3b286ef9ab
   cmake -S . -B build -DCMAKE_BUILD_TYPE=Release -DGGML_METAL=OFF -DGGML_NATIVE=ON
   cmake --build build --config Release -j 8
   ./build/bin/llama-server --version

   Expected result: Build succeeds and server version identifies pinned source. Stop if checkout or build fails.

Download the model

1. 1. Download the pinned Unsloth GGUF and Qwen template

   Verify both public immutable inputs. Qwen base/template and Unsloth quantization have separate revisions and authors. Use the explicit native Jinja template so non-thinking control does not depend on an uninspected embedded-template version. Stop on either hash mismatch.

   CommandCopy command

   cd "$HOME/qwen35-35ba3b-hosting"
   curl --fail --location --output models/Qwen3.5-35B-A3B-Q4_K_M.gguf 'https://huggingface.co/unsloth/Qwen3.5-35B-A3B-GGUF/resolve/bc014a17be43adabd7066b7a86075ff935c6a4e2/Qwen3.5-35B-A3B-Q4_K_M.gguf'
   curl --fail --location --output models/qwen35-chat-template.jinja 'https://huggingface.co/Qwen/Qwen3.5-35B-A3B/raw/59d61f3ce65a6d9863b86d2e96597125219dc754/chat_template.jinja'
   sha256sum --check <<'SHA256'
   3b46d1066bc91cc2d613e3bc22ce691dd77e6f0d33c9060690d24ce6de494375  models/Qwen3.5-35B-A3B-Q4_K_M.gguf
   a4aee8afcf2e0711942cf848899be66016f8d14a889ff9ede07bca099c28f715  models/qwen35-chat-template.jinja
   SHA256

   Expected result: Both files report OK. Full expert-weight GGUF is 22,016,023,168 bytes; native 35B-A3B template 7756 bytes. Stop on mismatch; no active-weight subset or sibling file is substituted.

Configure

1. 1. Create a private service credential

   Generate this only on your own host when following the guide. It is a local service password, not an OpenAI/provider credential. Keep it private and never paste it into tickets. Create it only when absent; preserve an existing key so a repeated setup does not silently rotate access. It is not created during this documentation run.

   CommandCopy command

   cd "$HOME/qwen35-35ba3b-hosting"
   umask 077
   if [ ! -e .api-key ]; then
     openssl rand -hex 32 > .api-key
   fi
   test -s .api-key || { echo 'Existing .api-key is empty; restore it before continuing.'; exit 1; }
   chmod 600 .api-key

   Expected result: A non-empty private .api-key file exists; existing key preserved. No secret is printed.

Start the model

1. 1. Start one CPU-only sequence

   Run the exact 40-layer Qwen3.5-35B-A3B MoE GGUF and native template. Pinned qwen35moe.cpp explicitly recognizes 40 layers as 35B_A3B and loads routed/shared experts plus Gated DeltaNet/full attention. CPU dispatch includes indexed expert matrix multiplication and Gated DeltaNet operations. This is documented support, not measured startup/fit. Keep one sequence, eight inference threads and 4,096 total context with room for 512 output tokens. Disable thinking with native enable_thinking=false and --reasoning off. No mmproj, tools or MTP.

   CommandCopy command

   cd "$HOME/qwen35-35ba3b-hosting"
   export QWEN35_35BA3B_SERVICE_KEY="$(cat .api-key)"
   ./llama.cpp/build/bin/llama-server -m models/Qwen3.5-35B-A3B-Q4_K_M.gguf --alias qwen35-35ba3b --host 127.0.0.1 --port 8080 --api-key "$QWEN35_35BA3B_SERVICE_KEY" --device none -ngl 0 -t 8 -c 4096 -np 1 --cache-type-k f16 --cache-type-v f16 --jinja --chat-template-file models/qwen35-chat-template.jinja --chat-template-kwargs '{"enable_thinking":false}' --reasoning off --reasoning-format none --skip-chat-parsing --temp 0.7 --top-p 0.8 --top-k 20 --min-p 0.0 --presence-penalty 1.5 --repeat-penalty 1.0

   Expected result: Model initializes and listener starts. Pass health and authenticated chat checks before relying on the service. Command-line credentials may be visible to privileged/local users; trusted host required.

Verify the setup

1. 1. Check readiness and authenticated text generation

   Use a second server terminal and the private service key. Send only plain text. Check final response against the supplied source passage and verify finish_reason is not length; no exact wording is guaranteed. Do not expect reasoning_content in raw mode. If extended thinking appears, non-thinking setup validation failed.

   CommandCopy command

   cd "$HOME/qwen35-35ba3b-hosting"
   export QWEN35_35BA3B_SERVICE_KEY="$(cat .api-key)"
   curl --fail http://127.0.0.1:8080/health
   curl --fail http://127.0.0.1:8080/v1/chat/completions -H "Authorization: Bearer $QWEN35_35BA3B_SERVICE_KEY" -H 'Content-Type: application/json' -d '{"model":"qwen35-35ba3b","messages":[{"role":"user","content":"Summarize this supplied passage in one sentence without adding facts: The community library opens at 9am on weekdays and closes at 6pm. On Saturday it opens at 10am and closes at 4pm. It is closed on Sunday."}],"max_tokens":512,"temperature":0.7,"top_k":20,"top_p":0.8,"min_p":0.0,"presence_penalty":1.5,"repeat_penalty":1.0}'
   curl -o /dev/null -s -w '%{http_code}\n' http://127.0.0.1:8080/v1/chat/completions -H 'Content-Type: application/json' -d '{"model":"qwen35-35ba3b","messages":[{"role":"user","content":"Hello"}]}'

   Expected result: Health HTTP 200 when ready; authenticated response contains non-empty choices[0].message.content with an accurate final summary and finish_reason not length, without a generated extended thinking trace. Unauthenticated chat returns HTTP 401. Verify facts yourself; output can be wrong.

Performance evidence

No compatible measurement available. Documented support and estimated memory do not establish response speed.

Limits and troubleshooting

• Documented configuration only. No weights, build, inference, full-fit, throughput or latency benchmark was executed by OpenTools.
• Unsloth’s complete Q4_K_M artifact is 22,016,023,168 bytes. The model card reports 35B total parameters and 3B active per token. All expert weights are part of the download; active parameter count does not establish storage, working RAM, compatibility or measured performance.
• This guide is text-only. No mmproj/projector is provided and image/video requests are outside its configuration. The model’s multimodal capability is not a claim that this text recipe validates vision.
• Qwen thinks by default. This guide pins its native template and disables thinking via enable_thinking=false plus --reasoning off. /think and /nothink are not officially supported soft switches. A 4K non-thinking recipe does not preserve/validate full thinking capability; the publisher advises at least 128K context for that purpose.
• The publisher advertises 262,144 native context with extension options. This recipe configures 4,096 total prompt and output, one active sequence and 512 validation output tokens. Longer context, thinking, concurrent users and another cache policy create different configurations.
• Use Qwen’s non-thinking general-task sampler (temperature 0.7, top-p 0.8, top-k 20, min-p 0, presence penalty 1.5 and repetition penalty 1.0). The model card gives conflicting non-thinking reasoning presets in two sections; this recipe avoids that task/mode and uses the consistent general-task preset.
• Summarization validation checks a short text request only. It does not establish coding, multilingual, long-context, retrieval, agentic, structured-output or multimodal quality; generated answers can be wrong.
• Operating system packages and build tool dependencies resolve when installed. Save tool versions/build logs with any actual execution. Local hardware is Intel Core i5-12400 with eight inference threads allocated; no exact-hardware benchmark is claimed.
• Loopback and a service key do not isolate the service from privileged local users. A shared/public service needs separately designed TLS, authentication and operations.
• The native checkpoint is Qwen3_5MoeForConditionalGeneration: 40 layers, hidden size 2,048, 256 routed experts with eight selected per token, plus a shared expert. It alternates three Gated DeltaNet layers per full-attention layer; linear key heads 16, value heads 32 and full-attention KV heads two. Do not copy dense 27B or LFM MoE cache assumptions.
• Native config mamba_ssm_dtype float32 does not establish llama.cpp recurrent-state precision or peak memory. F16 cache flags specify attention K/V only; routed/shared expert weights and hybrid recurrent state add distinct requirements.
• The unchanged 64 GiB / 80 GiB allocation is an author resource choice, not a measured fit result. No dense-model or sibling benchmark, confidence, speed or memory observation is transferred.

Download or checksum failure

Verify exact uppercase filename and pinned URL; remove only the incomplete dedicated model file and retry. Do not run a mismatching file.

Architecture unsupported or build fails

Confirm exact pinned source commit, Ubuntu compiler/CMake dependencies and GGUF. Do not substitute an older packaged llama.cpp silently.

Out of memory or context overflow

Stop server, free/provision RAM or reduce context and restart; shorten input/output within configured total. Allocation is not a measured guarantee; record any changed configuration.

Connection refused or HTTP 401

Check foreground server logs, 8080 binding, matching private key and Bearer header. Run the client on this same desktop and keep the loopback listener.

Unexpected thinking trace, template error or image request failure

Confirm both exact GGUF/template hashes, --jinja, enable_thinking=false and --reasoning off. Do not use /nothink as a substitute. Send only text; this recipe has no mmproj and does not support images/videos. If template or architecture loading fails, confirm exact pinned runtime and stop rather than substituting a different checkpoint.

MoE loader error or unexpectedly large download/memory use

Verify the exact full 35B-A3B file SHA and pinned qwen35moe runtime, not the dense qwen35 path. All routed/shared expert weights are needed. Do not use the 3B active label to size disk/RAM or substitute a dense sibling. Stop on load failure and retain logs; resource changes create a different configuration.

Stop the model

1. 1. Stop the service

   Press Ctrl+C in server terminal. Unset key in each shell. Confirm this desktop localhost:8080 no longer serves the model.

   CommandCopy command

   unset QWEN35_35BA3B_SERVICE_KEY

   Expected result: Server stops and localhost:8080 stops serving.

Uninstall

1. 1. Remove dedicated model, build and credential files

   Back up wanted logs/files first. Remove only this walkthrough directory after confirming its path. Shared apt packages can remain; do not remove other workloads.

   CommandCopy command

   rm -r -- "$HOME/qwen35-35ba3b-hosting"

   Expected result: Dedicated files and local service credential removed; shared system dependencies remain.

Related guides

• https://opentools.ai/llms/qwen35-35b-a3b

Exact model and requirements

Artifact:

Unsloth (GGUF quantization); Qwen (base model) / Qwen3.5-35B-A3B Q4_K_M, Unsloth derivative · checkpoint unsloth/Qwen3.5-35B-A3B-GGUF

Publisher revision:

bc014a17be43adabd7066b7a86075ff935c6a4e2

Artifact digest:

sha256:3b46d1066bc91cc2d613e3bc22ce691dd77e6f0d33c9060690d24ce6de494375

Download size:

20.5 GiB

Memory requirement (unknown):

Unknown — No exact CPU process peak for this complete MoE GGUF and hybrid cache configuration.

System RAM / disk:

Unknown / Unknown

Placement:

single device. One CPU host; no tensor/pipeline sharding. CPU-only execution; --device none and zero GPU layers. No disk or accelerator offload.

KV cache:

F16 attention K/V cache; Gated DeltaNet recurrent state is separately runtime-managed, with no claimed state precision or full-cache byte estimate.

Workload:

4,096 token context; Unknown; Unknown; concurrency 1; batch 1; cache unknown; repetitions Unknown.

Tokenizer:

Qwen3.5-35B-A3B GGUF tokenizer with separately pinned native 35B-A3B Jinja template · bc014a17be43adabd7066b7a86075ff935c6a4e2; template 59d61f3ce65a6d9863b86d2e96597125219dc754

Sampling / reasoning:

Qwen non-thinking general-task preset: temperature 0.7, top_p 0.8, top_k 20, min_p 0.0, presence_penalty 1.5, repeat_penalty 1.0. Validation max_tokens 512. Non-thinking text-only mode, independently supported by this 35B-A3B native template: enable_thinking=false and --reasoning off. No /nothink soft switch, vision projector, tool parser or MTP.

Runtime prerequisites:

Ubuntu 24.04 x86_64; See prerequisites above.

Official artifact · License

Qwen publishes Qwen3.5-35B-A3B under Apache License 2.0; Unsloth’s derivative metadata explicitly links that base and Apache-2.0 license. Read the full license, retain applicable license/copyright/attribution notices and mark modifications when redistributing covered work. Public access is not a substitute for license compliance. No provider inference subscription is required for the public artifact.

• qwen35-chat-template.jinja · 0 GiB · revision 59d61f3ce65a6d9863b86d2e96597125219dc754

• The complete 22,016,023,168-byte model plus attention/recurrent state, buffers, OS and setup files needs resources. Three billion active parameters does not shrink full model storage to a 3B model.
• No transformer-only KV estimate or dense/LFM sibling observation is used. Full working-memory fit, latency and speed remain unknown.

Install the runtime

1. 1. Build the pinned CPU server

   Install build dependencies and compile the exact official source revision. These packages are system shared dependencies; no CUDA/GPU driver is required. GGML_NATIVE=ON selects -march=native for this GNU/Linux x86 build, compiling for the host CPU. Build and run on the same machine; the binary uses the instruction set exposed by this r7i.2xlarge guest. Do not copy it to a different instance family without rebuilding.

   CommandCopy command

   sudo apt-get update
   sudo apt-get install -y build-essential cmake git curl ca-certificates openssl
   mkdir -p "$HOME/qwen35-35ba3b-hosting/models"
   cd "$HOME/qwen35-35ba3b-hosting"
   git clone https://github.com/ggml-org/llama.cpp.git
   cd llama.cpp
   git checkout --detach 620d2225cc436ab0b001e3200e590e3b286ef9ab
   cmake -S . -B build -DCMAKE_BUILD_TYPE=Release -DGGML_METAL=OFF -DGGML_NATIVE=ON
   cmake --build build --config Release -j 8
   ./build/bin/llama-server --version

   Expected result: Build succeeds and server version identifies pinned source. Stop if checkout or build fails.

Download the model

1. 1. Download the pinned Unsloth GGUF and Qwen template

   Verify both public immutable inputs. Qwen base/template and Unsloth quantization have separate revisions and authors. Use the explicit native Jinja template so non-thinking control does not depend on an uninspected embedded-template version. Stop on either hash mismatch.

   CommandCopy command

   cd "$HOME/qwen35-35ba3b-hosting"
   curl --fail --location --output models/Qwen3.5-35B-A3B-Q4_K_M.gguf 'https://huggingface.co/unsloth/Qwen3.5-35B-A3B-GGUF/resolve/bc014a17be43adabd7066b7a86075ff935c6a4e2/Qwen3.5-35B-A3B-Q4_K_M.gguf'
   curl --fail --location --output models/qwen35-chat-template.jinja 'https://huggingface.co/Qwen/Qwen3.5-35B-A3B/raw/59d61f3ce65a6d9863b86d2e96597125219dc754/chat_template.jinja'
   sha256sum --check <<'SHA256'
   3b46d1066bc91cc2d613e3bc22ce691dd77e6f0d33c9060690d24ce6de494375  models/Qwen3.5-35B-A3B-Q4_K_M.gguf
   a4aee8afcf2e0711942cf848899be66016f8d14a889ff9ede07bca099c28f715  models/qwen35-chat-template.jinja
   SHA256

   Expected result: Both files report OK. Full expert-weight GGUF is 22,016,023,168 bytes; native 35B-A3B template 7756 bytes. Stop on mismatch; no active-weight subset or sibling file is substituted.

Configure

1. 1. Create a private service credential

   Generate this only on your own host when following the guide. It is a local service password, not an OpenAI/provider credential. Keep it private and never paste it into tickets. Create it only when absent; preserve an existing key so a repeated setup does not silently rotate access. It is not created during this documentation run.

   CommandCopy command

   cd "$HOME/qwen35-35ba3b-hosting"
   umask 077
   if [ ! -e .api-key ]; then
     openssl rand -hex 32 > .api-key
   fi
   test -s .api-key || { echo 'Existing .api-key is empty; restore it before continuing.'; exit 1; }
   chmod 600 .api-key

   Expected result: A non-empty private .api-key file exists; existing key preserved. No secret is printed.

Start the model

1. 1. Start one CPU-only sequence

   Run the exact 40-layer Qwen3.5-35B-A3B MoE GGUF and native template. Pinned qwen35moe.cpp explicitly recognizes 40 layers as 35B_A3B and loads routed/shared experts plus Gated DeltaNet/full attention. CPU dispatch includes indexed expert matrix multiplication and Gated DeltaNet operations. This is documented support, not measured startup/fit. Keep one sequence, eight inference threads and 4,096 total context with room for 512 output tokens. Disable thinking with native enable_thinking=false and --reasoning off. No mmproj, tools or MTP.

   CommandCopy command

   cd "$HOME/qwen35-35ba3b-hosting"
   export QWEN35_35BA3B_SERVICE_KEY="$(cat .api-key)"
   ./llama.cpp/build/bin/llama-server -m models/Qwen3.5-35B-A3B-Q4_K_M.gguf --alias qwen35-35ba3b --host 127.0.0.1 --port 8080 --api-key "$QWEN35_35BA3B_SERVICE_KEY" --device none -ngl 0 -t 8 -c 4096 -np 1 --cache-type-k f16 --cache-type-v f16 --jinja --chat-template-file models/qwen35-chat-template.jinja --chat-template-kwargs '{"enable_thinking":false}' --reasoning off --reasoning-format none --skip-chat-parsing --temp 0.7 --top-p 0.8 --top-k 20 --min-p 0.0 --presence-penalty 1.5 --repeat-penalty 1.0

   Expected result: Model initializes and listener starts. Pass health and authenticated chat checks before relying on the service. Command-line credentials may be visible to privileged/local users; trusted host required.

Verify the setup

1. 1. Check readiness and authenticated text generation

   Use a second server terminal and the private service key. Send only plain text. Check final response against the supplied source passage and verify finish_reason is not length; no exact wording is guaranteed. Do not expect reasoning_content in raw mode. If extended thinking appears, non-thinking setup validation failed.

   CommandCopy command

   cd "$HOME/qwen35-35ba3b-hosting"
   export QWEN35_35BA3B_SERVICE_KEY="$(cat .api-key)"
   curl --fail http://127.0.0.1:8080/health
   curl --fail http://127.0.0.1:8080/v1/chat/completions -H "Authorization: Bearer $QWEN35_35BA3B_SERVICE_KEY" -H 'Content-Type: application/json' -d '{"model":"qwen35-35ba3b","messages":[{"role":"user","content":"Summarize this supplied passage in one sentence without adding facts: The community library opens at 9am on weekdays and closes at 6pm. On Saturday it opens at 10am and closes at 4pm. It is closed on Sunday."}],"max_tokens":512,"temperature":0.7,"top_k":20,"top_p":0.8,"min_p":0.0,"presence_penalty":1.5,"repeat_penalty":1.0}'
   curl -o /dev/null -s -w '%{http_code}\n' http://127.0.0.1:8080/v1/chat/completions -H 'Content-Type: application/json' -d '{"model":"qwen35-35ba3b","messages":[{"role":"user","content":"Hello"}]}'

   Expected result: Health HTTP 200 when ready; authenticated response contains non-empty choices[0].message.content with an accurate final summary and finish_reason not length, without a generated extended thinking trace. Unauthenticated chat returns HTTP 401. Verify facts yourself; output can be wrong.

Performance evidence

No compatible measurement available. Documented support and estimated memory do not establish response speed.

Limits and troubleshooting

• Documented configuration only. No weights, build, inference, full-fit, throughput or latency benchmark was executed by OpenTools.
• Unsloth’s complete Q4_K_M artifact is 22,016,023,168 bytes. The model card reports 35B total parameters and 3B active per token. All expert weights are part of the download; active parameter count does not establish storage, working RAM, compatibility or measured performance.
• This guide is text-only. No mmproj/projector is provided and image/video requests are outside its configuration. The model’s multimodal capability is not a claim that this text recipe validates vision.
• Qwen thinks by default. This guide pins its native template and disables thinking via enable_thinking=false plus --reasoning off. /think and /nothink are not officially supported soft switches. A 4K non-thinking recipe does not preserve/validate full thinking capability; the publisher advises at least 128K context for that purpose.
• The publisher advertises 262,144 native context with extension options. This recipe configures 4,096 total prompt and output, one active sequence and 512 validation output tokens. Longer context, thinking, concurrent users and another cache policy create different configurations.
• Use Qwen’s non-thinking general-task sampler (temperature 0.7, top-p 0.8, top-k 20, min-p 0, presence penalty 1.5 and repetition penalty 1.0). The model card gives conflicting non-thinking reasoning presets in two sections; this recipe avoids that task/mode and uses the consistent general-task preset.
• Summarization validation checks a short text request only. It does not establish coding, multilingual, long-context, retrieval, agentic, structured-output or multimodal quality; generated answers can be wrong.
• Operating system packages and build tool dependencies resolve when installed. Save tool versions/build logs with any actual execution. Cloud hardware is AWS r7i.2xlarge with eight vCPUs and 64 GiB host RAM; eight inference threads are allocated. No exact-instance benchmark is claimed.
• Loopback and a service key do not isolate the service from privileged local users. A shared/public service needs separately designed TLS, authentication and operations.
• The native checkpoint is Qwen3_5MoeForConditionalGeneration: 40 layers, hidden size 2,048, 256 routed experts with eight selected per token, plus a shared expert. It alternates three Gated DeltaNet layers per full-attention layer; linear key heads 16, value heads 32 and full-attention KV heads two. Do not copy dense 27B or LFM MoE cache assumptions.
• Native config mamba_ssm_dtype float32 does not establish llama.cpp recurrent-state precision or peak memory. F16 cache flags specify attention K/V only; routed/shared expert weights and hybrid recurrent state add distinct requirements.
• The unchanged 64 GiB / 80 GiB allocation is an author resource choice, not a measured fit result. No dense-model or sibling benchmark, confidence, speed or memory observation is transferred.

Download or checksum failure

Verify exact uppercase filename and pinned URL; remove only the incomplete dedicated model file and retry. Do not run a mismatching file.

Architecture unsupported or build fails

Confirm exact pinned source commit, Ubuntu compiler/CMake dependencies and GGUF. Do not substitute an older packaged llama.cpp silently.

Out of memory or context overflow

Stop server, free/provision RAM or reduce context and restart; shorten input/output within configured total. Allocation is not a measured guarantee; record any changed configuration.

Connection refused or HTTP 401

Check foreground server logs, 8080 binding, matching private key and Bearer header. Cloud client must have working SSH tunnel to the correct VM; do not expose the port to repair a tunnel.

Unexpected thinking trace, template error or image request failure

Confirm both exact GGUF/template hashes, --jinja, enable_thinking=false and --reasoning off. Do not use /nothink as a substitute. Send only text; this recipe has no mmproj and does not support images/videos. If template or architecture loading fails, confirm exact pinned runtime and stop rather than substituting a different checkpoint.

MoE loader error or unexpectedly large download/memory use

Verify the exact full 35B-A3B file SHA and pinned qwen35moe runtime, not the dense qwen35 path. All routed/shared expert weights are needed. Do not use the 3B active label to size disk/RAM or substitute a dense sibling. Stop on load failure and retain logs; resource changes create a different configuration.

Stop the model

1. 1. Stop the service

   Press Ctrl+C in server terminal. Unset key in each shell. Close the SSH tunnel with Ctrl+C. Stopping the process alone does not stop EC2 billing.

   CommandCopy command

   unset QWEN35_35BA3B_SERVICE_KEY

   Expected result: Server stops and localhost:8080 stops serving.

Uninstall

1. 1. Remove dedicated model, build and credential files

   Back up wanted logs/files first. Remove only this walkthrough directory after confirming its path. Shared apt packages can remain; do not remove other workloads.

   CommandCopy command

   rm -r -- "$HOME/qwen35-35ba3b-hosting"

   Expected result: Dedicated files and local service credential removed; shared system dependencies remain.

Shut down cloud resources

1. 1. End cloud billing resources

   In EC2 us-east-1, stop for later reuse or terminate this exact VM. Review Delete on termination for its root volume. Delete unwanted detached EBS volumes, snapshots and reserved/Elastic/public IP resources. Stopped VM storage and retained resources may still bill. Check Bills/Cost Explorer later for final usage.

   Expected result: Exact instance is stopped/terminated as intended; no unwanted attached/retained billed resources remain. Instance compute stops only after its lifecycle transition, not when llama-server stops.

Related guides

• https://opentools.ai/llms/qwen35-35b-a3b

Exact model and requirements

Artifact:

Unsloth (GGUF quantization); Qwen (base model) / Qwen3.5-35B-A3B Q4_K_M, Unsloth derivative · checkpoint unsloth/Qwen3.5-35B-A3B-GGUF

Publisher revision:

bc014a17be43adabd7066b7a86075ff935c6a4e2

Artifact digest:

sha256:3b46d1066bc91cc2d613e3bc22ce691dd77e6f0d33c9060690d24ce6de494375

Download size:

20.5 GiB

Memory requirement (unknown):

Unknown — No exact CPU process peak for this complete MoE GGUF and hybrid cache configuration.

System RAM / disk:

Unknown / Unknown

Placement:

single device. One CPU host; no tensor/pipeline sharding. CPU-only execution; --device none and zero GPU layers. No disk or accelerator offload. Vision projector explicitly remains CPU-only via --no-mmproj-offload.

KV cache:

F16 attention K/V cache; Gated DeltaNet recurrent state is separately runtime-managed, with no claimed state precision or full-cache byte estimate.

Workload:

4,096 token context; Unknown; Unknown; concurrency 1; batch 1; cache unknown; repetitions Unknown.

Tokenizer:

Qwen3.5-35B-A3B GGUF tokenizer with separately pinned native 35B-A3B Jinja template · bc014a17be43adabd7066b7a86075ff935c6a4e2; template 59d61f3ce65a6d9863b86d2e96597125219dc754

Sampling / reasoning:

Qwen non-thinking general-task preset: temperature 0.7, top_p 0.8, top_k 20, min_p 0.0, presence_penalty 1.5, repeat_penalty 1.0. Validation max_tokens 512. Non-thinking general image/text chat using exact35B native Jinja template enable_thinking=false and --reasoning off. Raw content without tool parser or MTP. One image; image/template/text/output share4096 context. No exact visual quality claimed.

Runtime prerequisites:

Ubuntu 24.04 x86_64; See prerequisites above.

Official artifact · License

Qwen publishes Qwen3.5-35B-A3B under Apache License 2.0; Unsloth’s derivative metadata explicitly links that base and Apache-2.0 license. Read the full license, retain applicable license/copyright/attribution notices and mark modifications when redistributing covered work. Public access is not a substitute for license compliance. No provider inference subscription is required for the public artifact.

• qwen35-chat-template.jinja · 0 GiB · revision 59d61f3ce65a6d9863b86d2e96597125219dc754
• mmproj-F16.gguf · 0.84 GiB · revision bc014a17be43adabd7066b7a86075ff935c6a4e2

• The complete 22,016,023,168-byte model plus attention/recurrent state, buffers, OS and setup files needs resources. Three billion active parameters does not shrink full model storage to a 3B model.
• No transformer-only KV estimate or dense/LFM sibling observation is used. Full working-memory fit, latency and speed remain unknown.
• Additional899283648-byte F16 vision projector plus decoded pixels, encoder/embedding buffers and warmup require RAM beyond language weights. Sparse active parameter count does not reduce full resident expert weight requirement.64GiB is allocation, not measured full fit.

Install the runtime

1. 1. Build the pinned CPU server

   Install build dependencies and compile the exact official source revision. These packages are system shared dependencies; no CUDA/GPU driver is required. GGML_NATIVE=ON selects -march=native for this GNU/Linux x86 build, compiling for the host CPU. Build and run on the same machine; no AVX-512 requirement is imposed on the i5-12400.

   CommandCopy command

   sudo apt-get update
   sudo apt-get install -y build-essential cmake git curl ca-certificates openssl python3
   mkdir -p "$HOME/qwen35-35ba3b-vision-hosting/models"
   cd "$HOME/qwen35-35ba3b-vision-hosting"
   git clone https://github.com/ggml-org/llama.cpp.git
   cd llama.cpp
   git checkout --detach 620d2225cc436ab0b001e3200e590e3b286ef9ab
   cmake -S . -B build -DCMAKE_BUILD_TYPE=Release -DGGML_METAL=OFF -DGGML_NATIVE=ON
   cmake --build build --config Release -j 8
   ./build/bin/llama-server --version

   Expected result: Build succeeds and server version identifies pinned source. Stop if checkout or build fails.

Download the model

1. 1. Download the pinned Unsloth GGUF and Qwen template

   Verify both public immutable inputs. Qwen base/template and Unsloth quantization have separate revisions and authors. Use the explicit native Jinja template so non-thinking control does not depend on an uninspected embedded-template version. Stop on either hash mismatch.

   CommandCopy command

   cd "$HOME/qwen35-35ba3b-vision-hosting"
   curl --fail --location --output models/Qwen3.5-35B-A3B-Q4_K_M.gguf 'https://huggingface.co/unsloth/Qwen3.5-35B-A3B-GGUF/resolve/bc014a17be43adabd7066b7a86075ff935c6a4e2/Qwen3.5-35B-A3B-Q4_K_M.gguf'
   curl --fail --location --output models/qwen35-chat-template.jinja 'https://huggingface.co/Qwen/Qwen3.5-35B-A3B/raw/59d61f3ce65a6d9863b86d2e96597125219dc754/chat_template.jinja'
   sha256sum --check <<'SHA256'
   3b46d1066bc91cc2d613e3bc22ce691dd77e6f0d33c9060690d24ce6de494375  models/Qwen3.5-35B-A3B-Q4_K_M.gguf
   a4aee8afcf2e0711942cf848899be66016f8d14a889ff9ede07bca099c28f715  models/qwen35-chat-template.jinja
   SHA256

   Expected result: Both files report OK. Full expert-weight GGUF is 22,016,023,168 bytes; native 35B-A3B template 7756 bytes. Stop on mismatch; no active-weight subset or sibling file is substituted.

2. 2. Download matching35B projector and tiny image fixture

   Same-revision Unsloth35B-A3B F16 projector required; checksum before use. Fixture is a pinned official runtime test image for private checking; no photograph redistribution rights asserted.

   CommandCopy command

   cd "$HOME/qwen35-35ba3b-vision-hosting"
   curl --fail --location --output models/mmproj-F16.gguf 'https://huggingface.co/unsloth/Qwen3.5-35B-A3B-GGUF/resolve/bc014a17be43adabd7066b7a86075ff935c6a4e2/mmproj-F16.gguf'
   curl --fail --location --output fixture.jpg 'https://raw.githubusercontent.com/ggml-org/llama.cpp/620d2225cc436ab0b001e3200e590e3b286ef9ab/tools/mtmd/test-1.jpeg'
   sha256sum --check <<'SHA256'
   a516ab92e8240da4734d68352bdfba84c16e830ee40010b8fac80d69c77272ff  models/mmproj-F16.gguf
   2dff664c0c8aaea18aff8cbe7e868845b775e90cdd7a0bac98df709b131deaa3  fixture.jpg
   SHA256

   Expected result: Projector and fixture report OK. Projector899283648 bytes; fixture124071 bytes. Stop on mismatch.

Configure

1. 1. Create a private service credential

   Generate this only on your own host when following the guide. It is a local service password, not an OpenAI/provider credential. Keep it private and never paste it into tickets. Create it only when absent; preserve an existing key so a repeated setup does not silently rotate access. It is not created during this documentation run.

   CommandCopy command

   cd "$HOME/qwen35-35ba3b-vision-hosting"
   umask 077
   if [ ! -e .api-key ]; then
     openssl rand -hex 32 > .api-key
   fi
   test -s .api-key || { echo 'Existing .api-key is empty; restore it before continuing.'; exit 1; }
   chmod 600 .api-key

   Expected result: A non-empty private .api-key file exists; existing key preserved. No secret is printed.

Start the model

1. 1. Start one CPU-only sequence

   Load the exact35B-A3B MoE language artifact plus matching same-revision F16 vision projector, all CPU. Pinned native converter explicitly maps Qwen3_5MoeForConditionalGeneration and Qwen/Qwen3.5-35B-A3B to QWEN3VL. Dynamic-resolution bicubic preprocessing uses model spatial merge metadata and server image min/max1024 flags. Actual count remains unmeasured; warmup may use larger vision buffers. One sequence,4096 total image/template/text/output context,512 output cap. Native35B template and consistent general-task sampler disable thinking; this is not a grounding benchmark.

   CommandCopy command

   cd "$HOME/qwen35-35ba3b-vision-hosting"
   export QWEN35_35BA3B_SERVICE_KEY="$(cat .api-key)"
   ./llama.cpp/build/bin/llama-server -m models/Qwen3.5-35B-A3B-Q4_K_M.gguf --mmproj models/mmproj-F16.gguf --no-mmproj-offload --image-min-tokens 1024 --image-max-tokens 1024 --alias qwen35-35ba3b --host 127.0.0.1 --port 8080 --api-key "$QWEN35_35BA3B_SERVICE_KEY" --device none -ngl 0 -t 8 -c 4096 -np 1 --cache-type-k f16 --cache-type-v f16 --jinja --chat-template-file models/qwen35-chat-template.jinja --chat-template-kwargs '{"enable_thinking":false}' --reasoning off --reasoning-format none --skip-chat-parsing --temp 0.7 --top-p 0.8 --top-k 20 --min-p 0.0 --presence-penalty 1.5 --repeat-penalty 1.0

   Expected result: Model initializes and listener starts. Pass health and authenticated chat checks before relying on the service. Command-line credentials may be visible to privileged/local users; trusted host required.

Verify the setup

1. 1. Check readiness and authenticated text generation

   Use a second server terminal and the private service key. Send only plain text. Check final response against the supplied source passage and verify finish_reason is not length; no exact wording is guaranteed. Do not expect reasoning_content in raw mode. If extended thinking appears, non-thinking setup validation failed.

   CommandCopy command

   cd "$HOME/qwen35-35ba3b-vision-hosting"
   export QWEN35_35BA3B_SERVICE_KEY="$(cat .api-key)"
   curl --fail http://127.0.0.1:8080/health
   curl --fail http://127.0.0.1:8080/v1/chat/completions -H "Authorization: Bearer $QWEN35_35BA3B_SERVICE_KEY" -H 'Content-Type: application/json' -d '{"model":"qwen35-35ba3b","messages":[{"role":"user","content":"Summarize this supplied passage in one sentence without adding facts: The community library opens at 9am on weekdays and closes at 6pm. On Saturday it opens at 10am and closes at 4pm. It is closed on Sunday."}],"max_tokens":512,"temperature":0.7,"top_k":20,"top_p":0.8,"min_p":0.0,"presence_penalty":1.5,"repeat_penalty":1.0}'
   curl -o /dev/null -s -w '%{http_code}\n' http://127.0.0.1:8080/v1/chat/completions -H 'Content-Type: application/json' -d '{"model":"qwen35-35ba3b","messages":[{"role":"user","content":"Hello"}]}'

   Expected result: Health HTTP 200 when ready; authenticated response contains non-empty choices[0].message.content with an accurate final summary and finish_reason not length, without a generated extended thinking trace. Unauthenticated chat returns HTTP 401. Verify facts yourself; output can be wrong.

2. 2. Check authenticated single-image understanding

   Run on the same server host (for cloud, inside the SSH shell). The JSON content array combines text and image_url with a local file encoded as data:image/jpeg;base64; no server-side remote image fetch or media-path flag is needed. Inspect content and finish_reason yourself. The fixture visibly shows a newspaper front page: The New York Times, MEN WALK ON MOON, astronauts landing/collecting rocks/planting a flag. A useful answer identifies the newspaper and the moon-landing headline; wording is not prescribed. Do not use unseen details or exact token counts as pass criteria.

   CommandCopy command

   cd "$HOME/qwen35-35ba3b-vision-hosting"
   export QWEN35_35BA3B_SERVICE_KEY="$(cat .api-key)"
   python3 - <<'PYREQUEST'
   import base64,json,pathlib
   image=base64.b64encode(pathlib.Path('fixture.jpg').read_bytes()).decode()
   request={'model':'qwen35-35ba3b','messages':[{'role':'user','content':[{'type':'text','text':'Identify the newspaper and transcribe its largest headline. Then briefly describe the front page.'},{'type':'image_url','image_url':{'url':'data:image/jpeg;base64,'+image}}]}],'max_tokens':512,'temperature':0.7,'top_p':0.8,'top_k':20,'min_p':0.0,'presence_penalty':1.5,'repeat_penalty':1.0}
   pathlib.Path('image-request.json').write_text(json.dumps(request))
   PYREQUEST
   curl --fail http://127.0.0.1:8080/v1/chat/completions -H "Authorization: Bearer $QWEN35_35BA3B_SERVICE_KEY" -H 'Content-Type: application/json' --data-binary @image-request.json

   Expected result: Authenticated HTTP 200 with nonempty choices[0].message.content identifying The New York Times and MEN WALK ON MOON; inspect grounded accuracy and finish_reason not length. No exact generated response promised. Image rejection, fabricated content, truncation or extended thinking fails this smoke check.

Performance evidence

No compatible measurement available. Documented support and estimated memory do not establish response speed.

Limits and troubleshooting

• Documented configuration only. No weights, build, inference, full-fit, throughput or latency benchmark was executed by OpenTools.
• Unsloth’s complete Q4_K_M artifact is 22,016,023,168 bytes. The model card reports 35B total parameters and 3B active per token. All expert weights are part of the download; active parameter count does not establish storage, working RAM, compatibility or measured performance.
• Qwen thinks by default. This guide pins its native template and disables thinking via enable_thinking=false plus --reasoning off. /think and /nothink are not officially supported soft switches. A 4K non-thinking recipe does not preserve/validate full thinking capability; the publisher advises at least 128K context for that purpose.
• The publisher advertises 262,144 native context with extension options. This recipe configures 4,096 total prompt and output, one active sequence and 512 validation output tokens. Longer context, thinking, concurrent users and another cache policy create different configurations.
• Use Qwen’s non-thinking general-task sampler (temperature 0.7, top-p 0.8, top-k 20, min-p 0, presence penalty 1.5 and repetition penalty 1.0). The model card gives conflicting non-thinking reasoning presets in two sections; this recipe avoids that task/mode and uses the consistent general-task preset.
• The supplied-passage summary and single-image fixture are bounded smoke checks; neither establishes coding, multilingual, long-context, retrieval, agentic, structured-output or general visual quality. Generated answers can be wrong.
• Operating system packages and build tool dependencies resolve when installed. Save tool versions/build logs with any actual execution. Local hardware is Intel Core i5-12400 with eight inference threads allocated; no exact-hardware benchmark is claimed.
• Loopback and a service key do not isolate the service from privileged local users. A shared/public service needs separately designed TLS, authentication and operations.
• The native checkpoint is Qwen3_5MoeForConditionalGeneration: 40 layers, hidden size 2,048, 256 routed experts with eight selected per token, plus a shared expert. It alternates three Gated DeltaNet layers per full-attention layer; linear key heads 16, value heads 32 and full-attention KV heads two. Do not copy dense 27B or LFM MoE cache assumptions.
• Native config mamba_ssm_dtype float32 does not establish llama.cpp recurrent-state precision or peak memory. F16 cache flags specify attention K/V only; routed/shared expert weights and hybrid recurrent state add distinct requirements.
• The unchanged 64 GiB / 80 GiB allocation is an author resource choice, not a measured fit result. No dense-model or sibling benchmark, confidence, speed or memory observation is transferred.
• Single-image description and the inherited supplied-passage text check are smoke checks only, not measured visual/OCR/grounding quality. The server may warm up larger vision buffers than the request; file bytes and the image-token ceiling do not prove fit. Actual prompt/image tokens, encoder memory, speed and quality remain unknown.
• No image preprocessing is delegated to Transformers: libmtmd uses its pinned dynamic-resolution QWEN3VL path, bicubic resize, model spatial merge metadata and CLI image token bounds. Do not transplant Hugging Face min_pixels/max_pixels request keys; this recipe sets server flags.

Download or checksum failure

Verify exact uppercase filename and pinned URL; remove only the incomplete dedicated model file and retry. Do not run a mismatching file.

Architecture unsupported or build fails

Confirm exact pinned source commit, Ubuntu compiler/CMake dependencies and GGUF. Do not substitute an older packaged llama.cpp silently.

Out of memory or context overflow

Stop server, free/provision RAM or reduce context and restart; shorten input/output within configured total. Allocation is not a measured guarantee; record any changed configuration.

Connection refused or HTTP 401

Check foreground server logs, 8080 binding, matching private key and Bearer header. Run the client on this same desktop and keep the loopback listener.

Unexpected thinking trace, template error or image request failure

Check exact35B language/template/projector hashes, --jinja enable_thinking=false, --reasoning off and --no-mmproj-offload. Confirm content array with text and image_url data URI. On embedding or architecture mismatch stop; never substitute a9B/27B projector.

MoE loader error or unexpectedly large download/memory use

Verify the exact full 35B-A3B file SHA and pinned qwen35moe runtime, not the dense qwen35 path. All routed/shared expert weights are needed. Do not use the 3B active label to size disk/RAM or substitute a dense sibling. Stop on load failure and retain logs; resource changes create a different configuration.

Stop the model

1. 1. Stop the service

   Press Ctrl+C in server terminal. Unset key in each shell. Confirm this desktop localhost:8080 no longer serves the model.

   CommandCopy command

   unset QWEN35_35BA3B_SERVICE_KEY

   Expected result: Server stops and localhost:8080 stops serving.

Uninstall

1. 1. Remove dedicated model, build and credential files

   Back up wanted logs/files first. Remove only this walkthrough directory after confirming its path. Shared apt packages can remain; do not remove other workloads.

   CommandCopy command

   rm -r -- "$HOME/qwen35-35ba3b-vision-hosting"

   Expected result: Dedicated files and local service credential removed; shared system dependencies remain.

Related guides

• https://opentools.ai/llms/qwen35-35b-a3b

Exact model and requirements

Artifact:

Unsloth (GGUF quantization); Qwen (base model) / Qwen3.5-35B-A3B Q4_K_M, Unsloth derivative · checkpoint unsloth/Qwen3.5-35B-A3B-GGUF

Publisher revision:

bc014a17be43adabd7066b7a86075ff935c6a4e2

Artifact digest:

sha256:3b46d1066bc91cc2d613e3bc22ce691dd77e6f0d33c9060690d24ce6de494375

Download size:

20.5 GiB

Memory requirement (unknown):

Unknown — No exact CPU process peak for this complete MoE GGUF and hybrid cache configuration.

System RAM / disk:

Unknown / Unknown

Placement:

single device. One CPU host; no tensor/pipeline sharding. CPU-only execution; --device none and zero GPU layers. No disk or accelerator offload. Vision projector explicitly remains CPU-only via --no-mmproj-offload.

KV cache:

F16 attention K/V cache; Gated DeltaNet recurrent state is separately runtime-managed, with no claimed state precision or full-cache byte estimate.

Workload:

4,096 token context; Unknown; Unknown; concurrency 1; batch 1; cache unknown; repetitions Unknown.

Tokenizer:

Qwen3.5-35B-A3B GGUF tokenizer with separately pinned native 35B-A3B Jinja template · bc014a17be43adabd7066b7a86075ff935c6a4e2; template 59d61f3ce65a6d9863b86d2e96597125219dc754

Sampling / reasoning:

Qwen non-thinking general-task preset: temperature 0.7, top_p 0.8, top_k 20, min_p 0.0, presence_penalty 1.5, repeat_penalty 1.0. Validation max_tokens 512. Non-thinking general image/text chat using exact35B native Jinja template enable_thinking=false and --reasoning off. Raw content without tool parser or MTP. One image; image/template/text/output share4096 context. No exact visual quality claimed.

Runtime prerequisites:

Ubuntu 24.04 x86_64; See prerequisites above.

Official artifact · License

Qwen publishes Qwen3.5-35B-A3B under Apache License 2.0; Unsloth’s derivative metadata explicitly links that base and Apache-2.0 license. Read the full license, retain applicable license/copyright/attribution notices and mark modifications when redistributing covered work. Public access is not a substitute for license compliance. No provider inference subscription is required for the public artifact.

• qwen35-chat-template.jinja · 0 GiB · revision 59d61f3ce65a6d9863b86d2e96597125219dc754
• mmproj-F16.gguf · 0.84 GiB · revision bc014a17be43adabd7066b7a86075ff935c6a4e2

• The complete 22,016,023,168-byte model plus attention/recurrent state, buffers, OS and setup files needs resources. Three billion active parameters does not shrink full model storage to a 3B model.
• No transformer-only KV estimate or dense/LFM sibling observation is used. Full working-memory fit, latency and speed remain unknown.
• Additional899283648-byte F16 vision projector plus decoded pixels, encoder/embedding buffers and warmup require RAM beyond language weights. Sparse active parameter count does not reduce full resident expert weight requirement.64GiB is allocation, not measured full fit.

Install the runtime

1. 1. Build the pinned CPU server

   Install build dependencies and compile the exact official source revision. These packages are system shared dependencies; no CUDA/GPU driver is required. GGML_NATIVE=ON selects -march=native for this GNU/Linux x86 build, compiling for the host CPU. Build and run on the same machine; the binary uses the instruction set exposed by this r7i.2xlarge guest. Do not copy it to a different instance family without rebuilding.

   CommandCopy command

   sudo apt-get update
   sudo apt-get install -y build-essential cmake git curl ca-certificates openssl python3
   mkdir -p "$HOME/qwen35-35ba3b-vision-hosting/models"
   cd "$HOME/qwen35-35ba3b-vision-hosting"
   git clone https://github.com/ggml-org/llama.cpp.git
   cd llama.cpp
   git checkout --detach 620d2225cc436ab0b001e3200e590e3b286ef9ab
   cmake -S . -B build -DCMAKE_BUILD_TYPE=Release -DGGML_METAL=OFF -DGGML_NATIVE=ON
   cmake --build build --config Release -j 8
   ./build/bin/llama-server --version

   Expected result: Build succeeds and server version identifies pinned source. Stop if checkout or build fails.

Download the model

1. 1. Download the pinned Unsloth GGUF and Qwen template

   Verify both public immutable inputs. Qwen base/template and Unsloth quantization have separate revisions and authors. Use the explicit native Jinja template so non-thinking control does not depend on an uninspected embedded-template version. Stop on either hash mismatch.

   CommandCopy command

   cd "$HOME/qwen35-35ba3b-vision-hosting"
   curl --fail --location --output models/Qwen3.5-35B-A3B-Q4_K_M.gguf 'https://huggingface.co/unsloth/Qwen3.5-35B-A3B-GGUF/resolve/bc014a17be43adabd7066b7a86075ff935c6a4e2/Qwen3.5-35B-A3B-Q4_K_M.gguf'
   curl --fail --location --output models/qwen35-chat-template.jinja 'https://huggingface.co/Qwen/Qwen3.5-35B-A3B/raw/59d61f3ce65a6d9863b86d2e96597125219dc754/chat_template.jinja'
   sha256sum --check <<'SHA256'
   3b46d1066bc91cc2d613e3bc22ce691dd77e6f0d33c9060690d24ce6de494375  models/Qwen3.5-35B-A3B-Q4_K_M.gguf
   a4aee8afcf2e0711942cf848899be66016f8d14a889ff9ede07bca099c28f715  models/qwen35-chat-template.jinja
   SHA256

   Expected result: Both files report OK. Full expert-weight GGUF is 22,016,023,168 bytes; native 35B-A3B template 7756 bytes. Stop on mismatch; no active-weight subset or sibling file is substituted.

2. 2. Download matching35B projector and tiny image fixture

   Same-revision Unsloth35B-A3B F16 projector required; checksum before use. Fixture is a pinned official runtime test image for private checking; no photograph redistribution rights asserted.

   CommandCopy command

   cd "$HOME/qwen35-35ba3b-vision-hosting"
   curl --fail --location --output models/mmproj-F16.gguf 'https://huggingface.co/unsloth/Qwen3.5-35B-A3B-GGUF/resolve/bc014a17be43adabd7066b7a86075ff935c6a4e2/mmproj-F16.gguf'
   curl --fail --location --output fixture.jpg 'https://raw.githubusercontent.com/ggml-org/llama.cpp/620d2225cc436ab0b001e3200e590e3b286ef9ab/tools/mtmd/test-1.jpeg'
   sha256sum --check <<'SHA256'
   a516ab92e8240da4734d68352bdfba84c16e830ee40010b8fac80d69c77272ff  models/mmproj-F16.gguf
   2dff664c0c8aaea18aff8cbe7e868845b775e90cdd7a0bac98df709b131deaa3  fixture.jpg
   SHA256

   Expected result: Projector and fixture report OK. Projector899283648 bytes; fixture124071 bytes. Stop on mismatch.

Configure

1. 1. Create a private service credential

   Generate this only on your own host when following the guide. It is a local service password, not an OpenAI/provider credential. Keep it private and never paste it into tickets. Create it only when absent; preserve an existing key so a repeated setup does not silently rotate access. It is not created during this documentation run.

   CommandCopy command

   cd "$HOME/qwen35-35ba3b-vision-hosting"
   umask 077
   if [ ! -e .api-key ]; then
     openssl rand -hex 32 > .api-key
   fi
   test -s .api-key || { echo 'Existing .api-key is empty; restore it before continuing.'; exit 1; }
   chmod 600 .api-key

   Expected result: A non-empty private .api-key file exists; existing key preserved. No secret is printed.

Start the model

1. 1. Start one CPU-only sequence

   Load the exact35B-A3B MoE language artifact plus matching same-revision F16 vision projector, all CPU. Pinned native converter explicitly maps Qwen3_5MoeForConditionalGeneration and Qwen/Qwen3.5-35B-A3B to QWEN3VL. Dynamic-resolution bicubic preprocessing uses model spatial merge metadata and server image min/max1024 flags. Actual count remains unmeasured; warmup may use larger vision buffers. One sequence,4096 total image/template/text/output context,512 output cap. Native35B template and consistent general-task sampler disable thinking; this is not a grounding benchmark.

   CommandCopy command

   cd "$HOME/qwen35-35ba3b-vision-hosting"
   export QWEN35_35BA3B_SERVICE_KEY="$(cat .api-key)"
   ./llama.cpp/build/bin/llama-server -m models/Qwen3.5-35B-A3B-Q4_K_M.gguf --mmproj models/mmproj-F16.gguf --no-mmproj-offload --image-min-tokens 1024 --image-max-tokens 1024 --alias qwen35-35ba3b --host 127.0.0.1 --port 8080 --api-key "$QWEN35_35BA3B_SERVICE_KEY" --device none -ngl 0 -t 8 -c 4096 -np 1 --cache-type-k f16 --cache-type-v f16 --jinja --chat-template-file models/qwen35-chat-template.jinja --chat-template-kwargs '{"enable_thinking":false}' --reasoning off --reasoning-format none --skip-chat-parsing --temp 0.7 --top-p 0.8 --top-k 20 --min-p 0.0 --presence-penalty 1.5 --repeat-penalty 1.0

   Expected result: Model initializes and listener starts. Pass health and authenticated chat checks before relying on the service. Command-line credentials may be visible to privileged/local users; trusted host required.

Verify the setup

1. 1. Check readiness and authenticated text generation

   Use a second server terminal and the private service key. Send only plain text. Check final response against the supplied source passage and verify finish_reason is not length; no exact wording is guaranteed. Do not expect reasoning_content in raw mode. If extended thinking appears, non-thinking setup validation failed.

   CommandCopy command

   cd "$HOME/qwen35-35ba3b-vision-hosting"
   export QWEN35_35BA3B_SERVICE_KEY="$(cat .api-key)"
   curl --fail http://127.0.0.1:8080/health
   curl --fail http://127.0.0.1:8080/v1/chat/completions -H "Authorization: Bearer $QWEN35_35BA3B_SERVICE_KEY" -H 'Content-Type: application/json' -d '{"model":"qwen35-35ba3b","messages":[{"role":"user","content":"Summarize this supplied passage in one sentence without adding facts: The community library opens at 9am on weekdays and closes at 6pm. On Saturday it opens at 10am and closes at 4pm. It is closed on Sunday."}],"max_tokens":512,"temperature":0.7,"top_k":20,"top_p":0.8,"min_p":0.0,"presence_penalty":1.5,"repeat_penalty":1.0}'
   curl -o /dev/null -s -w '%{http_code}\n' http://127.0.0.1:8080/v1/chat/completions -H 'Content-Type: application/json' -d '{"model":"qwen35-35ba3b","messages":[{"role":"user","content":"Hello"}]}'

   Expected result: Health HTTP 200 when ready; authenticated response contains non-empty choices[0].message.content with an accurate final summary and finish_reason not length, without a generated extended thinking trace. Unauthenticated chat returns HTTP 401. Verify facts yourself; output can be wrong.

2. 2. Check authenticated single-image understanding

   Run on the same server host (for cloud, inside the SSH shell). The JSON content array combines text and image_url with a local file encoded as data:image/jpeg;base64; no server-side remote image fetch or media-path flag is needed. Inspect content and finish_reason yourself. The fixture visibly shows a newspaper front page: The New York Times, MEN WALK ON MOON, astronauts landing/collecting rocks/planting a flag. A useful answer identifies the newspaper and the moon-landing headline; wording is not prescribed. Do not use unseen details or exact token counts as pass criteria.

   CommandCopy command

   cd "$HOME/qwen35-35ba3b-vision-hosting"
   export QWEN35_35BA3B_SERVICE_KEY="$(cat .api-key)"
   python3 - <<'PYREQUEST'
   import base64,json,pathlib
   image=base64.b64encode(pathlib.Path('fixture.jpg').read_bytes()).decode()
   request={'model':'qwen35-35ba3b','messages':[{'role':'user','content':[{'type':'text','text':'Identify the newspaper and transcribe its largest headline. Then briefly describe the front page.'},{'type':'image_url','image_url':{'url':'data:image/jpeg;base64,'+image}}]}],'max_tokens':512,'temperature':0.7,'top_p':0.8,'top_k':20,'min_p':0.0,'presence_penalty':1.5,'repeat_penalty':1.0}
   pathlib.Path('image-request.json').write_text(json.dumps(request))
   PYREQUEST
   curl --fail http://127.0.0.1:8080/v1/chat/completions -H "Authorization: Bearer $QWEN35_35BA3B_SERVICE_KEY" -H 'Content-Type: application/json' --data-binary @image-request.json

   Expected result: Authenticated HTTP 200 with nonempty choices[0].message.content identifying The New York Times and MEN WALK ON MOON; inspect grounded accuracy and finish_reason not length. No exact generated response promised. Image rejection, fabricated content, truncation or extended thinking fails this smoke check.

Performance evidence

No compatible measurement available. Documented support and estimated memory do not establish response speed.

Limits and troubleshooting

• Documented configuration only. No weights, build, inference, full-fit, throughput or latency benchmark was executed by OpenTools.
• Unsloth’s complete Q4_K_M artifact is 22,016,023,168 bytes. The model card reports 35B total parameters and 3B active per token. All expert weights are part of the download; active parameter count does not establish storage, working RAM, compatibility or measured performance.
• Qwen thinks by default. This guide pins its native template and disables thinking via enable_thinking=false plus --reasoning off. /think and /nothink are not officially supported soft switches. A 4K non-thinking recipe does not preserve/validate full thinking capability; the publisher advises at least 128K context for that purpose.
• The publisher advertises 262,144 native context with extension options. This recipe configures 4,096 total prompt and output, one active sequence and 512 validation output tokens. Longer context, thinking, concurrent users and another cache policy create different configurations.
• Use Qwen’s non-thinking general-task sampler (temperature 0.7, top-p 0.8, top-k 20, min-p 0, presence penalty 1.5 and repetition penalty 1.0). The model card gives conflicting non-thinking reasoning presets in two sections; this recipe avoids that task/mode and uses the consistent general-task preset.
• The supplied-passage summary and single-image fixture are bounded smoke checks; neither establishes coding, multilingual, long-context, retrieval, agentic, structured-output or general visual quality. Generated answers can be wrong.
• Operating system packages and build tool dependencies resolve when installed. Save tool versions/build logs with any actual execution. Cloud hardware is AWS r7i.2xlarge with eight vCPUs and 64 GiB host RAM; eight inference threads are allocated. No exact-instance benchmark is claimed.
• Loopback and a service key do not isolate the service from privileged local users. A shared/public service needs separately designed TLS, authentication and operations.
• The native checkpoint is Qwen3_5MoeForConditionalGeneration: 40 layers, hidden size 2,048, 256 routed experts with eight selected per token, plus a shared expert. It alternates three Gated DeltaNet layers per full-attention layer; linear key heads 16, value heads 32 and full-attention KV heads two. Do not copy dense 27B or LFM MoE cache assumptions.
• Native config mamba_ssm_dtype float32 does not establish llama.cpp recurrent-state precision or peak memory. F16 cache flags specify attention K/V only; routed/shared expert weights and hybrid recurrent state add distinct requirements.
• The unchanged 64 GiB / 80 GiB allocation is an author resource choice, not a measured fit result. No dense-model or sibling benchmark, confidence, speed or memory observation is transferred.
• Single-image description and the inherited supplied-passage text check are smoke checks only, not measured visual/OCR/grounding quality. The server may warm up larger vision buffers than the request; file bytes and the image-token ceiling do not prove fit. Actual prompt/image tokens, encoder memory, speed and quality remain unknown.
• No image preprocessing is delegated to Transformers: libmtmd uses its pinned dynamic-resolution QWEN3VL path, bicubic resize, model spatial merge metadata and CLI image token bounds. Do not transplant Hugging Face min_pixels/max_pixels request keys; this recipe sets server flags.

Download or checksum failure

Verify exact uppercase filename and pinned URL; remove only the incomplete dedicated model file and retry. Do not run a mismatching file.

Architecture unsupported or build fails

Confirm exact pinned source commit, Ubuntu compiler/CMake dependencies and GGUF. Do not substitute an older packaged llama.cpp silently.

Out of memory or context overflow

Stop server, free/provision RAM or reduce context and restart; shorten input/output within configured total. Allocation is not a measured guarantee; record any changed configuration.

Connection refused or HTTP 401

Check foreground server logs, 8080 binding, matching private key and Bearer header. Cloud client must have working SSH tunnel to the correct VM; do not expose the port to repair a tunnel.

Unexpected thinking trace, template error or image request failure

Check exact35B language/template/projector hashes, --jinja enable_thinking=false, --reasoning off and --no-mmproj-offload. Confirm content array with text and image_url data URI. On embedding or architecture mismatch stop; never substitute a9B/27B projector.

MoE loader error or unexpectedly large download/memory use

Verify the exact full 35B-A3B file SHA and pinned qwen35moe runtime, not the dense qwen35 path. All routed/shared expert weights are needed. Do not use the 3B active label to size disk/RAM or substitute a dense sibling. Stop on load failure and retain logs; resource changes create a different configuration.

Stop the model

1. 1. Stop the service

   Press Ctrl+C in server terminal. Unset key in each shell. Close the SSH tunnel with Ctrl+C. Stopping the process alone does not stop EC2 billing.

   CommandCopy command

   unset QWEN35_35BA3B_SERVICE_KEY

   Expected result: Server stops and localhost:8080 stops serving.

Uninstall

1. 1. Remove dedicated model, build and credential files

   Back up wanted logs/files first. Remove only this walkthrough directory after confirming its path. Shared apt packages can remain; do not remove other workloads.

   CommandCopy command

   rm -r -- "$HOME/qwen35-35ba3b-vision-hosting"

   Expected result: Dedicated files and local service credential removed; shared system dependencies remain.

Shut down cloud resources

1. 1. End cloud billing resources

   In EC2 us-east-1, stop for later reuse or terminate this exact VM. Review Delete on termination for its root volume. Delete unwanted detached EBS volumes, snapshots and reserved/Elastic/public IP resources. Stopped VM storage and retained resources may still bill. Check Bills/Cost Explorer later for final usage.

   Expected result: Exact instance is stopped/terminated as intended; no unwanted attached/retained billed resources remain. Instance compute stops only after its lifecycle transition, not when llama-server stops.

Related guides

• https://opentools.ai/llms/qwen35-35b-a3b

Exact model and requirements

Artifact:

Unsloth (GGUF quantization); Qwen (base model) / Qwen3.5-35B-A3B Q4_K_M, Unsloth derivative · checkpoint unsloth/Qwen3.5-35B-A3B-GGUF

Publisher revision:

bc014a17be43adabd7066b7a86075ff935c6a4e2

Artifact digest:

sha256:3b46d1066bc91cc2d613e3bc22ce691dd77e6f0d33c9060690d24ce6de494375

Download size:

20.5 GiB

Memory requirement (unknown):

Unknown — No exact CPU process peak for this complete MoE GGUF and hybrid cache configuration.

System RAM / disk:

Unknown / Unknown

Placement:

single device. One CPU host; no tensor/pipeline sharding. CPU-only execution; --device none and zero GPU layers. No disk or accelerator offload. Vision projector explicitly remains CPU-only via --no-mmproj-offload.

KV cache:

F16 attention K/V cache; Gated DeltaNet recurrent state is separately runtime-managed, with no claimed state precision or full-cache byte estimate.

Workload:

8,192 token context; Unknown; Unknown; concurrency 1; batch 1; cache unknown; repetitions Unknown.

Tokenizer:

Qwen3.5-35B-A3B GGUF tokenizer with separately pinned native 35B-A3B Jinja template · bc014a17be43adabd7066b7a86075ff935c6a4e2; template 59d61f3ce65a6d9863b86d2e96597125219dc754

Sampling / reasoning:

Qwen non-thinking general-task preset: temperature 0.7, top_p 0.8, top_k 20, min_p 0.0, presence_penalty 1.5, repeat_penalty 1.0. Validation max_tokens 512. Non-thinking native 35B-A3B template and sampler; temporal video via input_video.data with CPU language/projector and pinned FFmpeg. No audio transcript, tool executor or MTP.

Runtime prerequisites:

Ubuntu 24.04 x86_64; See prerequisites above.

Official artifact · License

Qwen publishes Qwen3.5-35B-A3B under Apache License 2.0; Unsloth’s derivative metadata explicitly links that base and Apache-2.0 license. Read the full license, retain applicable license/copyright/attribution notices and mark modifications when redistributing covered work. Public access is not a substitute for license compliance. No provider inference subscription is required for the public artifact.

• qwen35-chat-template.jinja · 0 GiB · revision 59d61f3ce65a6d9863b86d2e96597125219dc754
• mmproj-F16.gguf · 0.84 GiB · revision bc014a17be43adabd7066b7a86075ff935c6a4e2

• The complete 22,016,023,168-byte model plus attention/recurrent state, buffers, OS and setup files needs resources. Three billion active parameters does not shrink full model storage to a 3B model.
• No transformer-only KV estimate or dense/LFM sibling observation is used. Full working-memory fit, latency and speed remain unknown.
• Additional899283648-byte F16 vision projector plus decoded pixels, encoder/embedding buffers and warmup require RAM beyond language weights. Sparse active parameter count does not reduce full resident expert weight requirement.64GiB is allocation, not measured full fit.
• Video decoding RGB buffers, multiple visual embeddings, projector and MoE/recurrent state require working memory beyond model/projector file bytes. Actual peak and fit remain unknown.

Install the runtime

1. 1. Build the pinned CPU server

   Install build tools/Python, build FFmpeg n8.0.1 from its exact source commit into this walkthrough directory, then compile the exact llama.cpp CPU server with video and subprocess support. FFmpeg automatic external-library detection and network input are disabled for this local base64-file workflow; built-in H.264 decoding remains enabled. No new packages are installed during this research; these are reader commands. Preserve version/build logs.

   CommandCopy command

   sudo apt-get update
   sudo apt-get install -y build-essential cmake git curl ca-certificates openssl pkg-config python3
   mkdir -p "$HOME/qwen35-35ba3b-video-hosting/models"
   cd "$HOME/qwen35-35ba3b-video-hosting"
   git clone https://github.com/FFmpeg/FFmpeg.git ffmpeg-source
   git -C ffmpeg-source checkout --detach 894da5ca7d742e4429ffb2af534fcda0103ef593
   cd ffmpeg-source
   ./configure --prefix="$HOME/qwen35-35ba3b-video-hosting/ffmpeg-bin" --disable-x86asm --disable-doc --disable-debug --disable-autodetect --disable-network
   make -j 8
   make install
   cd ..
   export PATH="$HOME/qwen35-35ba3b-video-hosting/ffmpeg-bin/bin:$PATH"
   ffmpeg -version
   ffprobe -version
   git clone https://github.com/ggml-org/llama.cpp.git
   cd llama.cpp
   git checkout --detach 620d2225cc436ab0b001e3200e590e3b286ef9ab
   cmake -S . -B build -DCMAKE_BUILD_TYPE=Release -DGGML_METAL=OFF -DGGML_NATIVE=ON -DMTMD_VIDEO=ON -DLLAMA_SUBPROCESS=ON
   cmake --build build --config Release -j 8
   ./build/bin/llama-server --version

   Expected result: Pinned ffmpeg and ffprobe binaries report n8.0.1/source revision; llama-server build succeeds with MTMD_VIDEO and LLAMA_SUBPROCESS enabled. No model inference has run yet.

Download the model

1. 1. Download the pinned Unsloth GGUF and Qwen template

   Verify both public immutable inputs. Qwen base/template and Unsloth quantization have separate revisions and authors. Use the explicit native Jinja template so non-thinking control does not depend on an uninspected embedded-template version. Stop on either hash mismatch.

   CommandCopy command

   cd "$HOME/qwen35-35ba3b-video-hosting"
   curl --fail --location --output models/Qwen3.5-35B-A3B-Q4_K_M.gguf 'https://huggingface.co/unsloth/Qwen3.5-35B-A3B-GGUF/resolve/bc014a17be43adabd7066b7a86075ff935c6a4e2/Qwen3.5-35B-A3B-Q4_K_M.gguf'
   curl --fail --location --output models/qwen35-chat-template.jinja 'https://huggingface.co/Qwen/Qwen3.5-35B-A3B/raw/59d61f3ce65a6d9863b86d2e96597125219dc754/chat_template.jinja'
   sha256sum --check <<'SHA256'
   3b46d1066bc91cc2d613e3bc22ce691dd77e6f0d33c9060690d24ce6de494375  models/Qwen3.5-35B-A3B-Q4_K_M.gguf
   a4aee8afcf2e0711942cf848899be66016f8d14a889ff9ede07bca099c28f715  models/qwen35-chat-template.jinja
   SHA256

   Expected result: Both files report OK. Full expert-weight GGUF is 22,016,023,168 bytes; native 35B-A3B template 7756 bytes. Stop on mismatch; no active-weight subset or sibling file is substituted.

2. 2. Download matching35B projector and tiny image fixture

   Same-revision Unsloth35B-A3B F16 projector required; checksum before use. Fixture is a pinned official runtime test image for private checking; no photograph redistribution rights asserted.

   CommandCopy command

   cd "$HOME/qwen35-35ba3b-video-hosting"
   curl --fail --location --output models/mmproj-F16.gguf 'https://huggingface.co/unsloth/Qwen3.5-35B-A3B-GGUF/resolve/bc014a17be43adabd7066b7a86075ff935c6a4e2/mmproj-F16.gguf'
   curl --fail --location --output fixture.jpg 'https://raw.githubusercontent.com/ggml-org/llama.cpp/620d2225cc436ab0b001e3200e590e3b286ef9ab/tools/mtmd/test-1.jpeg'
   sha256sum --check <<'SHA256'
   a516ab92e8240da4734d68352bdfba84c16e830ee40010b8fac80d69c77272ff  models/mmproj-F16.gguf
   2dff664c0c8aaea18aff8cbe7e868845b775e90cdd7a0bac98df709b131deaa3  fixture.jpg
   SHA256

   Expected result: Projector and fixture report OK. Projector899283648 bytes; fixture124071 bytes. Stop on mismatch.

Configure

1. 1. Create a private service credential

   Generate this only on your own host when following the guide. It is a local service password, not an OpenAI/provider credential. Keep it private and never paste it into tickets. Create it only when absent; preserve an existing key so a repeated setup does not silently rotate access. It is not created during this documentation run.

   CommandCopy command

   cd "$HOME/qwen35-35ba3b-video-hosting"
   umask 077
   if [ ! -e .api-key ]; then
     openssl rand -hex 32 > .api-key
   fi
   test -s .api-key || { echo 'Existing .api-key is empty; restore it before continuing.'; exit 1; }
   chmod 600 .api-key

   Expected result: A non-empty private .api-key file exists; existing key preserved. No secret is printed.

Start the model

1. 1. Start one CPU-only sequence

   Start the exact 40-layer MoE language model and same-model projector on CPU, with the independently pinned native 35B-A3B template disabling thinking. Keep one sequence, eight inference threads, 8,192 total context and a 512-token output cap. Video uses 0.5 fps, nominal image min/max 1,024 tokens and encoding batches of 1,024. Conservative planning allowance 5 × 1,024 + 512 = 5,632 leaves 2,560 for text/template/timestamps; actual tokens/merge overhead are unknown. The runtime decodes the whole clip, so preflight is mandatory. Resampled timestamps are not exact original source times.

   CommandCopy command

   cd "$HOME/qwen35-35ba3b-video-hosting"
   export QWEN35_35BA3B_VIDEO_SERVICE_KEY="$(cat .api-key)"
   ./llama.cpp/build/bin/llama-server -m models/Qwen3.5-35B-A3B-Q4_K_M.gguf --mmproj models/mmproj-F16.gguf --no-mmproj-offload --image-min-tokens 1024 --image-max-tokens 1024 --alias qwen35-35ba3b --host 127.0.0.1 --port 8080 --api-key "$QWEN35_35BA3B_VIDEO_SERVICE_KEY" --device none -ngl 0 -t 8 -c 8192 -np 1 --cache-type-k f16 --cache-type-v f16 --jinja --chat-template-file models/qwen35-chat-template.jinja --chat-template-kwargs '{"enable_thinking":false}' --reasoning off --reasoning-format none --skip-chat-parsing --temp 0.7 --top-p 0.8 --top-k 20 --min-p 0.0 --presence-penalty 1.5 --repeat-penalty 1.0 --mmproj-device none --mtmd-batch-max-tokens 1024 --video-fps 0.5 --video-timestamp-interval 2000 --video-ffmpeg-dir "$HOME/qwen35-35ba3b-video-hosting/ffmpeg-bin/bin"

   Expected result: Model initializes and listener starts. Pass health and authenticated chat checks before relying on the service. Command-line credentials may be visible to privileged/local users; trusted host required.

Verify the setup

1. 1. Check readiness and authenticated text generation

   Use a second server terminal and the private service key. Send only plain text. Check final response against the supplied source passage and verify finish_reason is not length; no exact wording is guaranteed. Do not expect reasoning_content in raw mode. If extended thinking appears, non-thinking setup validation failed.

   CommandCopy command

   cd "$HOME/qwen35-35ba3b-video-hosting"
   export QWEN35_35BA3B_VIDEO_SERVICE_KEY="$(cat .api-key)"
   curl --fail http://127.0.0.1:8080/health
   curl --fail http://127.0.0.1:8080/v1/chat/completions -H "Authorization: Bearer $QWEN35_35BA3B_VIDEO_SERVICE_KEY" -H 'Content-Type: application/json' -d '{"model":"qwen35-35ba3b","messages":[{"role":"user","content":"Summarize this supplied passage in one sentence without adding facts: The community library opens at 9am on weekdays and closes at 6pm. On Saturday it opens at 10am and closes at 4pm. It is closed on Sunday."}],"max_tokens":512,"temperature":0.7,"top_k":20,"top_p":0.8,"min_p":0.0,"presence_penalty":1.5,"repeat_penalty":1.0}'
   curl -o /dev/null -s -w '%{http_code}\n' http://127.0.0.1:8080/v1/chat/completions -H 'Content-Type: application/json' -d '{"model":"qwen35-35ba3b","messages":[{"role":"user","content":"Hello"}]}'

   Expected result: Health HTTP 200 when ready; authenticated response contains non-empty choices[0].message.content with an accurate final summary and finish_reason not length, without a generated extended thinking trace. Unauthenticated chat returns HTTP 401. Verify facts yourself; output can be wrong.

2. 2. Confirm loaded video capability

   Health proves the process is ready; authenticated /props must also report both vision and video capability before video input. Stop if either is absent/false.

   CommandCopy command

   cd "$HOME/qwen35-35ba3b-video-hosting"
   export QWEN35_35BA3B_VIDEO_SERVICE_KEY="$(cat .api-key)"
   curl --fail http://127.0.0.1:8080/props -H "Authorization: Bearer $QWEN35_35BA3B_VIDEO_SERVICE_KEY" --output loaded-props.json
   python3 - <<'PYPROPS'
   import json,pathlib
   p=json.loads(pathlib.Path('loaded-props.json').read_text())
   assert p['modalities']['vision'] is True and p['modalities']['video'] is True
   print('Loaded vision and video capabilities confirmed')
   PYPROPS

   Expected result: Both modalities.vision and modalities.video are true. This is a required reader check, not an observed result from this authoring run.

3. 3. Verify all sampled frames before video submission

   Download and hash-check the whole immutable moving MP4. ffprobe verifies source metadata. Decode the entire file through the same fixed fps=0.500000 RGB24 filter as the native helper; count all bytes, with no -frames or artificial stop. Reject a changed source, oversized duration, partial frame or any count other than five. This is media decoding only, not model inference.

   CommandCopy command

   cd "$HOME/qwen35-35ba3b-video-hosting"
   export PATH="$HOME/qwen35-35ba3b-video-hosting/ffmpeg-bin/bin:$PATH"
   curl --fail --location --output fixture.mp4 'https://raw.githubusercontent.com/ggml-org/llama.cpp/620d2225cc436ab0b001e3200e590e3b286ef9ab/tools/mtmd/test-3.mp4'
   printf '%s  %s\n' '95a9776a0e7f8d6ab1ba67a31dbb09a4c04ffda3c853fd933624e3496fa970a5' 'fixture.mp4' | sha256sum --check
   ffprobe -v error -select_streams v:0 -show_entries stream=width,height,codec_name:format=duration -of json fixture.mp4 > fixture-metadata.json
   ffmpeg -nostdin -y -v error -i fixture.mp4 -vf fps=0.500000 -an -f rawvideo -pix_fmt rgb24 fixture-rgb.raw
   python3 - <<'PYPREFLIGHT'
   import json,pathlib
   m=json.loads(pathlib.Path('fixture-metadata.json').read_text())
   s=m['streams'][0]
   assert s['width']==854 and s['height']==358 and s['codec_name']=='h264'
   assert 10.02 <= float(m['format']['duration']) <= 10.04 and float(m['format']['duration']) <= 10.1
   size=pathlib.Path('fixture-rgb.raw').stat().st_size
   frame_bytes=854*358*3
   assert size % frame_bytes == 0, 'partial decoded frame'
   assert size // frame_bytes == 5, 'reject: expected exactly five sampled frames'
   print('Preflight OK: complete five-frame bounded input')
   PYPREFLIGHT

   Expected result: Fixture SHA OK; metadata matches; preflight reports exactly five complete sampled frames. If any assertion fails, stop before video submission. No generated frame limit hides additional decoded frames.

4. 4. Check temporal ordering with one short video

   Create input_video.data as raw MP4 base64 using the native API schema and send it only to this private local endpoint. The author visually inspected source frames, not a model output. Ask for observable scene order; do not require character names, dialogue or inferred motives. FFmpeg sampling can choose source frames near 1, 3, 5, 7 and 9 seconds; runtime timestamps are resampled positions and may differ from original frame times.

   CommandCopy command

   cd "$HOME/qwen35-35ba3b-video-hosting"
   python3 - <<'PYREQUEST'
   import base64,json,pathlib
   video=base64.b64encode(pathlib.Path('fixture.mp4').read_bytes()).decode()
   request={'model':'qwen35-35ba3b','messages':[{'role':'user','content':[{'type':'text','text':'Describe the beginning, middle and end of this short clip. Explain observable changes in the man’s position and his interaction with the chair. Do not infer dialogue, names or motives.'},{'type':'input_video','input_video':{'data':video}}]}],'max_tokens':512,'temperature':0.7,'top_p':0.8,'top_k':20,'min_p':0.0,'presence_penalty':1.5,'repeat_penalty':1.0}
   pathlib.Path('video-request.json').write_text(json.dumps(request))
   PYREQUEST
   export QWEN35_35BA3B_VIDEO_SERVICE_KEY="$(cat .api-key)"
   curl --fail http://127.0.0.1:8080/v1/chat/completions -H "Authorization: Bearer $QWEN35_35BA3B_VIDEO_SERVICE_KEY" -H 'Content-Type: application/json' --data-binary @video-request.json

   Expected result: Completed message.content correctly orders initially seated/caped man beside large gray character, later hand reaching/pulling chair-side handle, then man out of chair and standing/moving around the room. Inspect response against the clip/frames; no exact phrasing or exact source timestamps required. Wrong order, missing temporal change, context error or finish_reason length fails this narrow functional check. No model result is claimed here.

Performance evidence

No compatible measurement available. Documented support and estimated memory do not establish response speed.

Limits and troubleshooting

• Documented configuration only. No weights, build, inference, full-fit, throughput or latency benchmark was executed by OpenTools.
• Unsloth’s complete Q4_K_M artifact is 22,016,023,168 bytes. The model card reports 35B total parameters and 3B active per token. All expert weights are part of the download; active parameter count does not establish storage, working RAM, compatibility or measured performance.
• Use Qwen’s non-thinking general-task sampler (temperature 0.7, top-p 0.8, top-k 20, min-p 0, presence penalty 1.5 and repetition penalty 1.0). The model card gives conflicting non-thinking reasoning presets in two sections; this recipe avoids that task/mode and uses the consistent general-task preset.
• Operating system packages and build tool dependencies resolve when installed. Save tool versions/build logs with any actual execution. Local hardware is Intel Core i5-12400 with eight inference threads allocated; no exact-hardware benchmark is claimed.
• Loopback and a service key do not isolate the service from privileged local users. A shared/public service needs separately designed TLS, authentication and operations.
• The native checkpoint is Qwen3_5MoeForConditionalGeneration: 40 layers, hidden size 2,048, 256 routed experts with eight selected per token, plus a shared expert. It alternates three Gated DeltaNet layers per full-attention layer; linear key heads 16, value heads 32 and full-attention KV heads two. Do not copy dense 27B or LFM MoE cache assumptions.
• Native config mamba_ssm_dtype float32 does not establish llama.cpp recurrent-state precision or peak memory. F16 cache flags specify attention K/V only; routed/shared expert weights and hybrid recurrent state add distinct requirements.
• The unchanged 64 GiB / 80 GiB allocation is an author resource choice, not a measured fit result. No dense-model or sibling benchmark, confidence, speed or memory observation is transferred.
• No image preprocessing is delegated to Transformers: libmtmd uses its pinned dynamic-resolution QWEN3VL path, bicubic resize, model spatial merge metadata and CLI image token bounds. Do not transplant Hugging Face min_pixels/max_pixels request keys; this recipe sets server flags.
• This is a distinct temporal-video job: a real moving clip and ordered events, not a still image wrapped as video. It does not add audio understanding, video generation, arbitrary long video or validated multi-clip analysis.
• Source fixture is 10.03 seconds. Runtime decodes the whole input and has no hard request duration/frame limit, so preflight duration and all sampled frames are mandatory. 0.5 fps can miss fast events; the guide checks only this coarse seated-to-handle-to-standing sequence.
• The nominal five-frame image budget at 1,024 each plus 512 output fits within the 8K planning allocation with headroom, but actual image tokens, merge behavior, template and timestamp overhead are unmeasured. Runtime can temporally merge compatible QWEN3VL frames; no token discount or fit proof is inferred.
• Use non-thinking native Qwen template and general-task sampler. 8K does not exercise the publisher’s full advertised context or 128K thinking recommendation. No image/dense-sibling/video throughput or memory result is transferred.
• The official repository distributes the fixture, but media rights are unknown. This guide links it for private validation and does not authorize public rehosting. Source-frame visual inspection is human evidence, not executed inference.
• This 35B-A3B MoE has 35B total and 3B active parameters; the full routed/shared-expert model plus video encoder/state/buffers is required. Dense 9B/27B video support, memory or quality is not transferred; only the exact shared runtime/fixture controls are reused.
• The 64 GiB RAM and 80 GiB disk values are resource allocations, not measured video fit or minimums. No temporal-merge token discount, model inference, exact latency or throughput claim is made.

Download or checksum failure

Verify exact uppercase filename and pinned URL; remove only the incomplete dedicated model file and retry. Do not run a mismatching file.

Architecture unsupported or build fails

Confirm exact pinned source commit, Ubuntu compiler/CMake dependencies and GGUF. Do not substitute an older packaged llama.cpp silently.

Out of memory or context overflow

Stop server, free/provision RAM or reduce context and restart; shorten input/output within configured total. Allocation is not a measured guarantee; record any changed configuration.

Connection refused or HTTP 401

Check foreground server logs, 8080 binding, matching private key and Bearer header. Run the client on this same desktop and keep the loopback listener.

MoE loader error or unexpectedly large download/memory use

Verify the exact full 35B-A3B file SHA and pinned qwen35moe runtime, not the dense qwen35 path. All routed/shared expert weights are needed. Do not use the 3B active label to size disk/RAM or substitute a dense sibling. Stop on load failure and retain logs; resource changes create a different configuration.

Video not supported, FFmpeg error, too many frames or wrong event order

Confirm video/subprocess build options, pinned ffmpeg and ffprobe in --video-ffmpeg-dir, exact MP4 SHA and full five-frame preflight. Use native input_video.data, not image_url or a plain path. Never ignore a failed duration/count check or cap decoding to hide extra frames. Wrong/truncated ordering fails validation; inspect sampled-frame limitations before changing fps/context, which changes this configuration.

Stop the model

1. 1. Stop the service

   Press Ctrl+C in server terminal. Unset key in each shell. Confirm this desktop localhost:8080 no longer serves the model.

   CommandCopy command

   unset QWEN35_35BA3B_VIDEO_SERVICE_KEY

   Expected result: Server stops and localhost:8080 stops serving.

Uninstall

1. 1. Remove dedicated model, build and credential files

   Back up wanted logs/files first. Remove only this walkthrough directory after confirming its path. Shared apt packages can remain; do not remove other workloads.

   CommandCopy command

   rm -r -- "$HOME/qwen35-35ba3b-video-hosting"

   Expected result: Dedicated files and local service credential removed; shared system dependencies remain.

Related guides

• https://opentools.ai/llms/qwen35-35b-a3b

Exact model and requirements

Artifact:

Unsloth (GGUF quantization); Qwen (base model) / Qwen3.5-35B-A3B Q4_K_M, Unsloth derivative · checkpoint unsloth/Qwen3.5-35B-A3B-GGUF

Publisher revision:

bc014a17be43adabd7066b7a86075ff935c6a4e2

Artifact digest:

sha256:3b46d1066bc91cc2d613e3bc22ce691dd77e6f0d33c9060690d24ce6de494375

Download size:

20.5 GiB

Memory requirement (unknown):

Unknown — No exact CPU process peak for this complete MoE GGUF and hybrid cache configuration.

System RAM / disk:

Unknown / Unknown

Placement:

single device. One CPU host; no tensor/pipeline sharding. CPU-only execution; --device none and zero GPU layers. No disk or accelerator offload. Vision projector explicitly remains CPU-only via --no-mmproj-offload.

KV cache:

F16 attention K/V cache; Gated DeltaNet recurrent state is separately runtime-managed, with no claimed state precision or full-cache byte estimate.

Workload:

8,192 token context; Unknown; Unknown; concurrency 1; batch 1; cache unknown; repetitions Unknown.

Tokenizer:

Qwen3.5-35B-A3B GGUF tokenizer with separately pinned native 35B-A3B Jinja template · bc014a17be43adabd7066b7a86075ff935c6a4e2; template 59d61f3ce65a6d9863b86d2e96597125219dc754

Sampling / reasoning:

Qwen non-thinking general-task preset: temperature 0.7, top_p 0.8, top_k 20, min_p 0.0, presence_penalty 1.5, repeat_penalty 1.0. Validation max_tokens 512. Non-thinking native 35B-A3B template and sampler; temporal video via input_video.data with CPU language/projector and pinned FFmpeg. No audio transcript, tool executor or MTP.

Runtime prerequisites:

Ubuntu 24.04 x86_64; See prerequisites above.

Official artifact · License

Qwen publishes Qwen3.5-35B-A3B under Apache License 2.0; Unsloth’s derivative metadata explicitly links that base and Apache-2.0 license. Read the full license, retain applicable license/copyright/attribution notices and mark modifications when redistributing covered work. Public access is not a substitute for license compliance. No provider inference subscription is required for the public artifact.

• qwen35-chat-template.jinja · 0 GiB · revision 59d61f3ce65a6d9863b86d2e96597125219dc754
• mmproj-F16.gguf · 0.84 GiB · revision bc014a17be43adabd7066b7a86075ff935c6a4e2

• The complete 22,016,023,168-byte model plus attention/recurrent state, buffers, OS and setup files needs resources. Three billion active parameters does not shrink full model storage to a 3B model.
• No transformer-only KV estimate or dense/LFM sibling observation is used. Full working-memory fit, latency and speed remain unknown.
• Additional899283648-byte F16 vision projector plus decoded pixels, encoder/embedding buffers and warmup require RAM beyond language weights. Sparse active parameter count does not reduce full resident expert weight requirement.64GiB is allocation, not measured full fit.
• Video decoding RGB buffers, multiple visual embeddings, projector and MoE/recurrent state require working memory beyond model/projector file bytes. Actual peak and fit remain unknown.

Install the runtime

1. 1. Build the pinned CPU server

   Install build tools/Python, build FFmpeg n8.0.1 from its exact source commit into this walkthrough directory, then compile the exact llama.cpp CPU server with video and subprocess support. FFmpeg automatic external-library detection and network input are disabled for this local base64-file workflow; built-in H.264 decoding remains enabled. No new packages are installed during this research; these are reader commands. Preserve version/build logs.

   CommandCopy command

   sudo apt-get update
   sudo apt-get install -y build-essential cmake git curl ca-certificates openssl pkg-config python3
   mkdir -p "$HOME/qwen35-35ba3b-video-hosting/models"
   cd "$HOME/qwen35-35ba3b-video-hosting"
   git clone https://github.com/FFmpeg/FFmpeg.git ffmpeg-source
   git -C ffmpeg-source checkout --detach 894da5ca7d742e4429ffb2af534fcda0103ef593
   cd ffmpeg-source
   ./configure --prefix="$HOME/qwen35-35ba3b-video-hosting/ffmpeg-bin" --disable-x86asm --disable-doc --disable-debug --disable-autodetect --disable-network
   make -j 8
   make install
   cd ..
   export PATH="$HOME/qwen35-35ba3b-video-hosting/ffmpeg-bin/bin:$PATH"
   ffmpeg -version
   ffprobe -version
   git clone https://github.com/ggml-org/llama.cpp.git
   cd llama.cpp
   git checkout --detach 620d2225cc436ab0b001e3200e590e3b286ef9ab
   cmake -S . -B build -DCMAKE_BUILD_TYPE=Release -DGGML_METAL=OFF -DGGML_NATIVE=ON -DMTMD_VIDEO=ON -DLLAMA_SUBPROCESS=ON
   cmake --build build --config Release -j 8
   ./build/bin/llama-server --version

   Expected result: Pinned ffmpeg and ffprobe binaries report n8.0.1/source revision; llama-server build succeeds with MTMD_VIDEO and LLAMA_SUBPROCESS enabled. No model inference has run yet.

Download the model

1. 1. Download the pinned Unsloth GGUF and Qwen template

   Verify both public immutable inputs. Qwen base/template and Unsloth quantization have separate revisions and authors. Use the explicit native Jinja template so non-thinking control does not depend on an uninspected embedded-template version. Stop on either hash mismatch.

   CommandCopy command

   cd "$HOME/qwen35-35ba3b-video-hosting"
   curl --fail --location --output models/Qwen3.5-35B-A3B-Q4_K_M.gguf 'https://huggingface.co/unsloth/Qwen3.5-35B-A3B-GGUF/resolve/bc014a17be43adabd7066b7a86075ff935c6a4e2/Qwen3.5-35B-A3B-Q4_K_M.gguf'
   curl --fail --location --output models/qwen35-chat-template.jinja 'https://huggingface.co/Qwen/Qwen3.5-35B-A3B/raw/59d61f3ce65a6d9863b86d2e96597125219dc754/chat_template.jinja'
   sha256sum --check <<'SHA256'
   3b46d1066bc91cc2d613e3bc22ce691dd77e6f0d33c9060690d24ce6de494375  models/Qwen3.5-35B-A3B-Q4_K_M.gguf
   a4aee8afcf2e0711942cf848899be66016f8d14a889ff9ede07bca099c28f715  models/qwen35-chat-template.jinja
   SHA256

   Expected result: Both files report OK. Full expert-weight GGUF is 22,016,023,168 bytes; native 35B-A3B template 7756 bytes. Stop on mismatch; no active-weight subset or sibling file is substituted.

2. 2. Download matching35B projector and tiny image fixture

   Same-revision Unsloth35B-A3B F16 projector required; checksum before use. Fixture is a pinned official runtime test image for private checking; no photograph redistribution rights asserted.

   CommandCopy command

   cd "$HOME/qwen35-35ba3b-video-hosting"
   curl --fail --location --output models/mmproj-F16.gguf 'https://huggingface.co/unsloth/Qwen3.5-35B-A3B-GGUF/resolve/bc014a17be43adabd7066b7a86075ff935c6a4e2/mmproj-F16.gguf'
   curl --fail --location --output fixture.jpg 'https://raw.githubusercontent.com/ggml-org/llama.cpp/620d2225cc436ab0b001e3200e590e3b286ef9ab/tools/mtmd/test-1.jpeg'
   sha256sum --check <<'SHA256'
   a516ab92e8240da4734d68352bdfba84c16e830ee40010b8fac80d69c77272ff  models/mmproj-F16.gguf
   2dff664c0c8aaea18aff8cbe7e868845b775e90cdd7a0bac98df709b131deaa3  fixture.jpg
   SHA256

   Expected result: Projector and fixture report OK. Projector899283648 bytes; fixture124071 bytes. Stop on mismatch.

Configure

1. 1. Create a private service credential

   Generate this only on your own host when following the guide. It is a local service password, not an OpenAI/provider credential. Keep it private and never paste it into tickets. Create it only when absent; preserve an existing key so a repeated setup does not silently rotate access. It is not created during this documentation run.

   CommandCopy command

   cd "$HOME/qwen35-35ba3b-video-hosting"
   umask 077
   if [ ! -e .api-key ]; then
     openssl rand -hex 32 > .api-key
   fi
   test -s .api-key || { echo 'Existing .api-key is empty; restore it before continuing.'; exit 1; }
   chmod 600 .api-key

   Expected result: A non-empty private .api-key file exists; existing key preserved. No secret is printed.

Start the model

1. 1. Start one CPU-only sequence

   Start the exact 40-layer MoE language model and same-model projector on CPU, with the independently pinned native 35B-A3B template disabling thinking. Keep one sequence, eight inference threads, 8,192 total context and a 512-token output cap. Video uses 0.5 fps, nominal image min/max 1,024 tokens and encoding batches of 1,024. Conservative planning allowance 5 × 1,024 + 512 = 5,632 leaves 2,560 for text/template/timestamps; actual tokens/merge overhead are unknown. The runtime decodes the whole clip, so preflight is mandatory. Resampled timestamps are not exact original source times.

   CommandCopy command

   cd "$HOME/qwen35-35ba3b-video-hosting"
   export QWEN35_35BA3B_VIDEO_SERVICE_KEY="$(cat .api-key)"
   ./llama.cpp/build/bin/llama-server -m models/Qwen3.5-35B-A3B-Q4_K_M.gguf --mmproj models/mmproj-F16.gguf --no-mmproj-offload --image-min-tokens 1024 --image-max-tokens 1024 --alias qwen35-35ba3b --host 127.0.0.1 --port 8080 --api-key "$QWEN35_35BA3B_VIDEO_SERVICE_KEY" --device none -ngl 0 -t 8 -c 8192 -np 1 --cache-type-k f16 --cache-type-v f16 --jinja --chat-template-file models/qwen35-chat-template.jinja --chat-template-kwargs '{"enable_thinking":false}' --reasoning off --reasoning-format none --skip-chat-parsing --temp 0.7 --top-p 0.8 --top-k 20 --min-p 0.0 --presence-penalty 1.5 --repeat-penalty 1.0 --mmproj-device none --mtmd-batch-max-tokens 1024 --video-fps 0.5 --video-timestamp-interval 2000 --video-ffmpeg-dir "$HOME/qwen35-35ba3b-video-hosting/ffmpeg-bin/bin"

   Expected result: Model initializes and listener starts. Pass health and authenticated chat checks before relying on the service. Command-line credentials may be visible to privileged/local users; trusted host required.

Verify the setup

1. 1. Check readiness and authenticated text generation

   Use a second server terminal and the private service key. Send only plain text. Check final response against the supplied source passage and verify finish_reason is not length; no exact wording is guaranteed. Do not expect reasoning_content in raw mode. If extended thinking appears, non-thinking setup validation failed.

   CommandCopy command

   cd "$HOME/qwen35-35ba3b-video-hosting"
   export QWEN35_35BA3B_VIDEO_SERVICE_KEY="$(cat .api-key)"
   curl --fail http://127.0.0.1:8080/health
   curl --fail http://127.0.0.1:8080/v1/chat/completions -H "Authorization: Bearer $QWEN35_35BA3B_VIDEO_SERVICE_KEY" -H 'Content-Type: application/json' -d '{"model":"qwen35-35ba3b","messages":[{"role":"user","content":"Summarize this supplied passage in one sentence without adding facts: The community library opens at 9am on weekdays and closes at 6pm. On Saturday it opens at 10am and closes at 4pm. It is closed on Sunday."}],"max_tokens":512,"temperature":0.7,"top_k":20,"top_p":0.8,"min_p":0.0,"presence_penalty":1.5,"repeat_penalty":1.0}'
   curl -o /dev/null -s -w '%{http_code}\n' http://127.0.0.1:8080/v1/chat/completions -H 'Content-Type: application/json' -d '{"model":"qwen35-35ba3b","messages":[{"role":"user","content":"Hello"}]}'

   Expected result: Health HTTP 200 when ready; authenticated response contains non-empty choices[0].message.content with an accurate final summary and finish_reason not length, without a generated extended thinking trace. Unauthenticated chat returns HTTP 401. Verify facts yourself; output can be wrong.

2. 2. Confirm loaded video capability

   Health proves the process is ready; authenticated /props must also report both vision and video capability before video input. Stop if either is absent/false.

   CommandCopy command

   cd "$HOME/qwen35-35ba3b-video-hosting"
   export QWEN35_35BA3B_VIDEO_SERVICE_KEY="$(cat .api-key)"
   curl --fail http://127.0.0.1:8080/props -H "Authorization: Bearer $QWEN35_35BA3B_VIDEO_SERVICE_KEY" --output loaded-props.json
   python3 - <<'PYPROPS'
   import json,pathlib
   p=json.loads(pathlib.Path('loaded-props.json').read_text())
   assert p['modalities']['vision'] is True and p['modalities']['video'] is True
   print('Loaded vision and video capabilities confirmed')
   PYPROPS

   Expected result: Both modalities.vision and modalities.video are true. This is a required reader check, not an observed result from this authoring run.

3. 3. Verify all sampled frames before video submission

   Download and hash-check the whole immutable moving MP4. ffprobe verifies source metadata. Decode the entire file through the same fixed fps=0.500000 RGB24 filter as the native helper; count all bytes, with no -frames or artificial stop. Reject a changed source, oversized duration, partial frame or any count other than five. This is media decoding only, not model inference.

   CommandCopy command

   cd "$HOME/qwen35-35ba3b-video-hosting"
   export PATH="$HOME/qwen35-35ba3b-video-hosting/ffmpeg-bin/bin:$PATH"
   curl --fail --location --output fixture.mp4 'https://raw.githubusercontent.com/ggml-org/llama.cpp/620d2225cc436ab0b001e3200e590e3b286ef9ab/tools/mtmd/test-3.mp4'
   printf '%s  %s\n' '95a9776a0e7f8d6ab1ba67a31dbb09a4c04ffda3c853fd933624e3496fa970a5' 'fixture.mp4' | sha256sum --check
   ffprobe -v error -select_streams v:0 -show_entries stream=width,height,codec_name:format=duration -of json fixture.mp4 > fixture-metadata.json
   ffmpeg -nostdin -y -v error -i fixture.mp4 -vf fps=0.500000 -an -f rawvideo -pix_fmt rgb24 fixture-rgb.raw
   python3 - <<'PYPREFLIGHT'
   import json,pathlib
   m=json.loads(pathlib.Path('fixture-metadata.json').read_text())
   s=m['streams'][0]
   assert s['width']==854 and s['height']==358 and s['codec_name']=='h264'
   assert 10.02 <= float(m['format']['duration']) <= 10.04 and float(m['format']['duration']) <= 10.1
   size=pathlib.Path('fixture-rgb.raw').stat().st_size
   frame_bytes=854*358*3
   assert size % frame_bytes == 0, 'partial decoded frame'
   assert size // frame_bytes == 5, 'reject: expected exactly five sampled frames'
   print('Preflight OK: complete five-frame bounded input')
   PYPREFLIGHT

   Expected result: Fixture SHA OK; metadata matches; preflight reports exactly five complete sampled frames. If any assertion fails, stop before video submission. No generated frame limit hides additional decoded frames.

4. 4. Check temporal ordering with one short video

   Create input_video.data as raw MP4 base64 using the native API schema and send it only to this private local endpoint. The author visually inspected source frames, not a model output. Ask for observable scene order; do not require character names, dialogue or inferred motives. FFmpeg sampling can choose source frames near 1, 3, 5, 7 and 9 seconds; runtime timestamps are resampled positions and may differ from original frame times.

   CommandCopy command

   cd "$HOME/qwen35-35ba3b-video-hosting"
   python3 - <<'PYREQUEST'
   import base64,json,pathlib
   video=base64.b64encode(pathlib.Path('fixture.mp4').read_bytes()).decode()
   request={'model':'qwen35-35ba3b','messages':[{'role':'user','content':[{'type':'text','text':'Describe the beginning, middle and end of this short clip. Explain observable changes in the man’s position and his interaction with the chair. Do not infer dialogue, names or motives.'},{'type':'input_video','input_video':{'data':video}}]}],'max_tokens':512,'temperature':0.7,'top_p':0.8,'top_k':20,'min_p':0.0,'presence_penalty':1.5,'repeat_penalty':1.0}
   pathlib.Path('video-request.json').write_text(json.dumps(request))
   PYREQUEST
   export QWEN35_35BA3B_VIDEO_SERVICE_KEY="$(cat .api-key)"
   curl --fail http://127.0.0.1:8080/v1/chat/completions -H "Authorization: Bearer $QWEN35_35BA3B_VIDEO_SERVICE_KEY" -H 'Content-Type: application/json' --data-binary @video-request.json

   Expected result: Completed message.content correctly orders initially seated/caped man beside large gray character, later hand reaching/pulling chair-side handle, then man out of chair and standing/moving around the room. Inspect response against the clip/frames; no exact phrasing or exact source timestamps required. Wrong order, missing temporal change, context error or finish_reason length fails this narrow functional check. No model result is claimed here.

Performance evidence

No compatible measurement available. Documented support and estimated memory do not establish response speed.

Limits and troubleshooting

• Documented configuration only. No weights, build, inference, full-fit, throughput or latency benchmark was executed by OpenTools.
• Unsloth’s complete Q4_K_M artifact is 22,016,023,168 bytes. The model card reports 35B total parameters and 3B active per token. All expert weights are part of the download; active parameter count does not establish storage, working RAM, compatibility or measured performance.
• Use Qwen’s non-thinking general-task sampler (temperature 0.7, top-p 0.8, top-k 20, min-p 0, presence penalty 1.5 and repetition penalty 1.0). The model card gives conflicting non-thinking reasoning presets in two sections; this recipe avoids that task/mode and uses the consistent general-task preset.
• Operating system packages and build tool dependencies resolve when installed. Save tool versions/build logs with any actual execution. Cloud hardware is AWS r7i.2xlarge with eight vCPUs and 64 GiB host RAM; eight inference threads are allocated. No exact-instance benchmark is claimed.
• Loopback and a service key do not isolate the service from privileged local users. A shared/public service needs separately designed TLS, authentication and operations.
• The native checkpoint is Qwen3_5MoeForConditionalGeneration: 40 layers, hidden size 2,048, 256 routed experts with eight selected per token, plus a shared expert. It alternates three Gated DeltaNet layers per full-attention layer; linear key heads 16, value heads 32 and full-attention KV heads two. Do not copy dense 27B or LFM MoE cache assumptions.
• Native config mamba_ssm_dtype float32 does not establish llama.cpp recurrent-state precision or peak memory. F16 cache flags specify attention K/V only; routed/shared expert weights and hybrid recurrent state add distinct requirements.
• The unchanged 64 GiB / 80 GiB allocation is an author resource choice, not a measured fit result. No dense-model or sibling benchmark, confidence, speed or memory observation is transferred.
• No image preprocessing is delegated to Transformers: libmtmd uses its pinned dynamic-resolution QWEN3VL path, bicubic resize, model spatial merge metadata and CLI image token bounds. Do not transplant Hugging Face min_pixels/max_pixels request keys; this recipe sets server flags.
• This is a distinct temporal-video job: a real moving clip and ordered events, not a still image wrapped as video. It does not add audio understanding, video generation, arbitrary long video or validated multi-clip analysis.
• Source fixture is 10.03 seconds. Runtime decodes the whole input and has no hard request duration/frame limit, so preflight duration and all sampled frames are mandatory. 0.5 fps can miss fast events; the guide checks only this coarse seated-to-handle-to-standing sequence.
• The nominal five-frame image budget at 1,024 each plus 512 output fits within the 8K planning allocation with headroom, but actual image tokens, merge behavior, template and timestamp overhead are unmeasured. Runtime can temporally merge compatible QWEN3VL frames; no token discount or fit proof is inferred.
• Use non-thinking native Qwen template and general-task sampler. 8K does not exercise the publisher’s full advertised context or 128K thinking recommendation. No image/dense-sibling/video throughput or memory result is transferred.
• The official repository distributes the fixture, but media rights are unknown. This guide links it for private validation and does not authorize public rehosting. Source-frame visual inspection is human evidence, not executed inference.
• This 35B-A3B MoE has 35B total and 3B active parameters; the full routed/shared-expert model plus video encoder/state/buffers is required. Dense 9B/27B video support, memory or quality is not transferred; only the exact shared runtime/fixture controls are reused.
• The 64 GiB RAM and 80 GiB disk values are resource allocations, not measured video fit or minimums. No temporal-merge token discount, model inference, exact latency or throughput claim is made.

Download or checksum failure

Verify exact uppercase filename and pinned URL; remove only the incomplete dedicated model file and retry. Do not run a mismatching file.

Architecture unsupported or build fails

Confirm exact pinned source commit, Ubuntu compiler/CMake dependencies and GGUF. Do not substitute an older packaged llama.cpp silently.

Out of memory or context overflow

Stop server, free/provision RAM or reduce context and restart; shorten input/output within configured total. Allocation is not a measured guarantee; record any changed configuration.

Connection refused or HTTP 401

Check foreground server logs, 8080 binding, matching private key and Bearer header. Cloud client must have working SSH tunnel to the correct VM; do not expose the port to repair a tunnel.

MoE loader error or unexpectedly large download/memory use

Verify the exact full 35B-A3B file SHA and pinned qwen35moe runtime, not the dense qwen35 path. All routed/shared expert weights are needed. Do not use the 3B active label to size disk/RAM or substitute a dense sibling. Stop on load failure and retain logs; resource changes create a different configuration.

Video not supported, FFmpeg error, too many frames or wrong event order

Confirm video/subprocess build options, pinned ffmpeg and ffprobe in --video-ffmpeg-dir, exact MP4 SHA and full five-frame preflight. Use native input_video.data, not image_url or a plain path. Never ignore a failed duration/count check or cap decoding to hide extra frames. Wrong/truncated ordering fails validation; inspect sampled-frame limitations before changing fps/context, which changes this configuration.

Stop the model

1. 1. Stop the service

   Press Ctrl+C in server terminal. Unset key in each shell. Close the SSH tunnel with Ctrl+C. Stopping the process alone does not stop EC2 billing.

   CommandCopy command

   unset QWEN35_35BA3B_VIDEO_SERVICE_KEY

   Expected result: Server stops and localhost:8080 stops serving.

Uninstall

1. 1. Remove dedicated model, build and credential files

   Back up wanted logs/files first. Remove only this walkthrough directory after confirming its path. Shared apt packages can remain; do not remove other workloads.

   CommandCopy command

   rm -r -- "$HOME/qwen35-35ba3b-video-hosting"

   Expected result: Dedicated files and local service credential removed; shared system dependencies remain.

Shut down cloud resources

1. 1. End cloud billing resources

   In EC2 us-east-1, stop for later reuse or terminate this exact VM. Review Delete on termination for its root volume. Delete unwanted detached EBS volumes, snapshots and reserved/Elastic/public IP resources. Stopped VM storage and retained resources may still bill. Check Bills/Cost Explorer later for final usage.

   Expected result: Exact instance is stopped/terminated as intended; no unwanted attached/retained billed resources remain. Instance compute stops only after its lifecycle transition, not when llama-server stops.

Related guides

• https://opentools.ai/llms/qwen35-35b-a3b