Qwen3.5-27B

Exact model and requirements

Artifact:

Unsloth (GGUF quantization); Qwen (base model) / Qwen3.5-27B Q4_K_M, Unsloth derivative · checkpoint unsloth/Qwen3.5-27B-GGUF

Publisher revision:

3221f178a6b842d04f1fb42f1c413534adcc0a6a

Artifact digest:

sha256:84b5f7f112156d63836a01a69dc3f11a6ba63b10a23b8ca7a7efaf52d5a2d806

Download size:

15.59 GiB

Memory requirement (unknown):

Unknown — No exact 27B hybrid-state/process peak memory evidence for this configuration.

System RAM / disk:

Unknown / Unknown

Placement:

single device. One CPU host; no tensor/pipeline sharding. CPU-only execution; --device none and zero GPU layers. No disk or accelerator offload.

KV cache:

F16 attention K/V cache; Gated DeltaNet recurrent state is separately runtime-managed, with no claimed state precision or full-cache byte estimate.

Workload:

4,096 token context; Unknown; Unknown; concurrency 1; batch 1; cache unknown; repetitions Unknown.

Tokenizer:

Qwen3.5-27B GGUF tokenizer with separately pinned Qwen 27B native Jinja template · 3221f178a6b842d04f1fb42f1c413534adcc0a6a; template fc05daec18b0a78c049392ed2e771dde82bdf654

Sampling / reasoning:

Qwen non-thinking general-task preset: temperature 0.7, top_p 0.8, top_k 20, min_p 0.0, presence_penalty 1.5, repeat_penalty 1.0. Validation max_tokens 512. Non-thinking text-only mode: native Qwen Jinja template, enable_thinking=false, --reasoning off. No /think or /nothink soft switch. Raw output mode; no tool parser, MTP drafter or vision projector.

Runtime prerequisites:

Ubuntu 24.04 x86_64; See prerequisites above.

Official artifact · License

Qwen publishes Qwen3.5-27B under Apache License 2.0; Unsloth’s derivative metadata explicitly names this base model, declares Apache-2.0 and links its license. Read the full license, retain applicable license/copyright/attribution notices and mark modifications when redistributing covered work. Public ungated metadata is not a substitute for license compliance. No provider API subscription is required for the public download.

• qwen35-chat-template.jinja · 0 GiB · revision fc05daec18b0a78c049392ed2e771dde82bdf654

• The full 16,740,812,704-byte GGUF plus attention cache, recurrent state, buffers, OS and setup artifacts need resources. File bytes alone do not prove fit.
• 64 GiB RAM and 80 GiB disk were selected for headroom, not as exact required-memory estimates. No transformer-only KV formula or sibling observations are used.

Install the runtime

1. 1. Build the pinned CPU server

   Install build dependencies and compile the exact official source revision. These packages are system shared dependencies; no CUDA/GPU driver is required. GGML_NATIVE=ON selects -march=native for this GNU/Linux x86 build, compiling for the host CPU. Build and run on the same machine; no AVX-512 requirement is imposed on the i5-12400.

   CommandCopy command

   sudo apt-get update
   sudo apt-get install -y build-essential cmake git curl ca-certificates openssl
   mkdir -p "$HOME/qwen35-27b-hosting/models"
   cd "$HOME/qwen35-27b-hosting"
   git clone https://github.com/ggml-org/llama.cpp.git
   cd llama.cpp
   git checkout --detach 620d2225cc436ab0b001e3200e590e3b286ef9ab
   cmake -S . -B build -DCMAKE_BUILD_TYPE=Release -DGGML_METAL=OFF -DGGML_NATIVE=ON
   cmake --build build --config Release -j 8
   ./build/bin/llama-server --version

   Expected result: Build succeeds and server version identifies pinned source. Stop if checkout or build fails.

Download the model

1. 1. Download the pinned Unsloth GGUF and Qwen template

   Verify both public immutable inputs. Qwen base/template and Unsloth quantization have separate revisions and authors. Use the explicit native Jinja template so non-thinking control does not depend on an uninspected embedded-template version. Stop on either hash mismatch.

   CommandCopy command

   cd "$HOME/qwen35-27b-hosting"
   curl --fail --location --output models/Qwen3.5-27B-Q4_K_M.gguf 'https://huggingface.co/unsloth/Qwen3.5-27B-GGUF/resolve/3221f178a6b842d04f1fb42f1c413534adcc0a6a/Qwen3.5-27B-Q4_K_M.gguf'
   curl --fail --location --output models/qwen35-chat-template.jinja 'https://huggingface.co/Qwen/Qwen3.5-27B/raw/fc05daec18b0a78c049392ed2e771dde82bdf654/chat_template.jinja'
   sha256sum --check <<'SHA256'
   84b5f7f112156d63836a01a69dc3f11a6ba63b10a23b8ca7a7efaf52d5a2d806  models/Qwen3.5-27B-Q4_K_M.gguf
   a4aee8afcf2e0711942cf848899be66016f8d14a889ff9ede07bca099c28f715  models/qwen35-chat-template.jinja
   SHA256

   Expected result: Both files report OK. Exact Q4_K_M GGUF is 16,740,812,704 bytes and native 27B template 7756 bytes. No weights from another size and no projector are substituted.

Configure

1. 1. Create a private service credential

   Generate this only on your own host when following the guide. It is a local service password, not an OpenAI/provider credential. Keep it private and never paste it into tickets. Create it only when absent; preserve an existing key so a repeated setup does not silently rotate access. It is not created during this documentation run.

   CommandCopy command

   cd "$HOME/qwen35-27b-hosting"
   umask 077
   if [ ! -e .api-key ]; then
     openssl rand -hex 32 > .api-key
   fi
   test -s .api-key || { echo 'Existing .api-key is empty; restore it before continuing.'; exit 1; }
   chmod 600 .api-key

   Expected result: A non-empty private .api-key file exists; existing key preserved. No secret is printed.

Start the model

1. 1. Start one CPU-only sequence

   Serve the exact 64-layer Qwen3.5 dense/hybrid 27B GGUF with the separately pinned native 27B template. The pinned qwen35 runtime explicitly recognizes 64 layers as its 27B variant, and CPU dispatch includes Gated DeltaNet and SSM operations. Documented support does not prove this allocation starts or fits. Use enable_thinking=false and --reasoning off; no soft /nothink switch. Keep one sequence, 4,096 total context tokens, and room for 512 output tokens. No mmproj, tools or MTP.

   CommandCopy command

   cd "$HOME/qwen35-27b-hosting"
   export QWEN35_27B_SERVICE_KEY="$(cat .api-key)"
   ./llama.cpp/build/bin/llama-server -m models/Qwen3.5-27B-Q4_K_M.gguf --alias qwen35-27b --host 127.0.0.1 --port 8080 --api-key "$QWEN35_27B_SERVICE_KEY" --device none -ngl 0 -t 8 -c 4096 -np 1 --cache-type-k f16 --cache-type-v f16 --jinja --chat-template-file models/qwen35-chat-template.jinja --chat-template-kwargs '{"enable_thinking":false}' --reasoning off --reasoning-format none --skip-chat-parsing --temp 0.7 --top-p 0.8 --top-k 20 --min-p 0.0 --presence-penalty 1.5 --repeat-penalty 1.0

   Expected result: Model initializes and listener starts. Pass health and authenticated chat checks before relying on the service. Command-line credentials may be visible to privileged/local users; trusted host required.

Verify the setup

1. 1. Check readiness and authenticated text generation

   Use a second server terminal and the private service key. Send only plain text. Check final response against the supplied source passage and verify finish_reason is not length; no exact wording is guaranteed. Do not expect reasoning_content in raw mode. If extended thinking appears, non-thinking setup validation failed.

   CommandCopy command

   cd "$HOME/qwen35-27b-hosting"
   export QWEN35_27B_SERVICE_KEY="$(cat .api-key)"
   curl --fail http://127.0.0.1:8080/health
   curl --fail http://127.0.0.1:8080/v1/chat/completions -H "Authorization: Bearer $QWEN35_27B_SERVICE_KEY" -H 'Content-Type: application/json' -d '{"model":"qwen35-27b","messages":[{"role":"user","content":"Summarize this supplied passage in one sentence without adding facts: The community library opens at 9am on weekdays and closes at 6pm. On Saturday it opens at 10am and closes at 4pm. It is closed on Sunday."}],"max_tokens":512,"temperature":0.7,"top_k":20,"top_p":0.8,"min_p":0.0,"presence_penalty":1.5,"repeat_penalty":1.0}'
   curl -o /dev/null -s -w '%{http_code}\n' http://127.0.0.1:8080/v1/chat/completions -H 'Content-Type: application/json' -d '{"model":"qwen35-27b","messages":[{"role":"user","content":"Hello"}]}'

   Expected result: Health HTTP 200 when ready; authenticated response contains non-empty choices[0].message.content with an accurate final summary and finish_reason not length, without a generated extended thinking trace. Unauthenticated chat returns HTTP 401. Verify facts yourself; output can be wrong.

Performance evidence

No compatible measurement available. Documented support and estimated memory do not establish response speed.

Limits and troubleshooting

• Documented configuration only. No weights, build, inference, full-fit, throughput or latency benchmark was executed by OpenTools.
• Exact Unsloth Q4_K_M artifact is 16,740,812,704 bytes, derived from Qwen/Qwen3.5-27B at separate revisions. Quantization does not guarantee full-precision quality. No weight-size-only memory fit is claimed.
• The native 27B checkpoint is a dense language backbone with a vision encoder, 64 layers and hidden size 5,120. Its config alternates three Gated DeltaNet layers per full-attention layer; it has 16 linear key heads, 48 linear value heads and four full-attention KV heads. Family MoE claims do not make this checkpoint sparse. Hybrid recurrent state and full-attention cache must not be reduced to a transformer-only memory formula.
• This guide is text-only. No mmproj/projector is provided and image/video requests are outside its configuration. The model’s multimodal capability is not a claim that this text recipe validates vision.
• Qwen thinks by default. This guide pins its native template and disables thinking via enable_thinking=false plus --reasoning off. /think and /nothink are not officially supported soft switches. A 4K non-thinking recipe does not preserve/validate full thinking capability; the publisher advises at least 128K context for that purpose.
• The publisher advertises 262,144 native context with extension options. This recipe configures 4,096 total prompt and output, one active sequence and 512 validation output tokens. Longer context, thinking, concurrent users and another cache policy create different configurations.
• Use Qwen’s non-thinking general-task sampler (temperature 0.7, top-p 0.8, top-k 20, min-p 0, presence penalty 1.5 and repetition penalty 1.0). The model card gives conflicting non-thinking reasoning presets in two sections; this recipe avoids that task/mode and uses the consistent general-task preset.
• Summarization validation checks a short text request only. It does not establish coding, multilingual, long-context, retrieval, agentic, structured-output or multimodal quality; generated answers can be wrong.
• Operating system packages and build tool dependencies resolve when installed. Save tool versions/build logs with any actual execution. Local hardware is Intel Core i5-12400 with eight inference threads allocated; no exact-hardware benchmark is claimed.
• Loopback and a service key do not isolate the service from privileged local users. A shared/public service needs separately designed TLS, authentication and operations.
• The native Transformers config names mamba_ssm_dtype float32, but that does not establish llama.cpp recurrent-state precision or peak memory. This recipe explicitly sets only F16 attention K/V cache; runtime-managed recurrent state remains unmeasured.
• The four-thread, 32 GiB 9B sibling setup is not evidence for this eight-thread, 64 GiB 27B configuration. No throughput, memory or quality observation is transferred from it.

Download or checksum failure

Verify exact uppercase filename and pinned URL; remove only the incomplete dedicated model file and retry. Do not run a mismatching file.

Architecture unsupported or build fails

Confirm exact pinned source commit, Ubuntu compiler/CMake dependencies and GGUF. Do not substitute an older packaged llama.cpp silently.

Out of memory or context overflow

Stop server, free/provision RAM or reduce context and restart; shorten input/output within configured total. Allocation is not a measured guarantee; record any changed configuration.

Connection refused or HTTP 401

Check foreground server logs, 8080 binding, matching private key and Bearer header. Run the client on this same desktop and keep the loopback listener.

Unexpected thinking trace, template error or image request failure

Confirm both exact GGUF/template hashes, --jinja, enable_thinking=false and --reasoning off. Do not use /nothink as a substitute. Send only text; this recipe has no mmproj and does not support images/videos. If template or architecture loading fails, confirm exact pinned runtime and stop rather than substituting a different checkpoint.

Stop the model

1. 1. Stop the service

   Press Ctrl+C in server terminal. Unset key in each shell. Confirm this desktop localhost:8080 no longer serves the model.

   CommandCopy command

   unset QWEN35_27B_SERVICE_KEY

   Expected result: Server stops and localhost:8080 stops serving.

Uninstall

1. 1. Remove dedicated model, build and credential files

   Back up wanted logs/files first. Remove only this walkthrough directory after confirming its path. Shared apt packages can remain; do not remove other workloads.

   CommandCopy command

   rm -r -- "$HOME/qwen35-27b-hosting"

   Expected result: Dedicated files and local service credential removed; shared system dependencies remain.

Related guides

• https://opentools.ai/llms/qwen35-27b

Exact model and requirements

Artifact:

Unsloth (GGUF quantization); Qwen (base model) / Qwen3.5-27B Q4_K_M, Unsloth derivative · checkpoint unsloth/Qwen3.5-27B-GGUF

Publisher revision:

3221f178a6b842d04f1fb42f1c413534adcc0a6a

Artifact digest:

sha256:84b5f7f112156d63836a01a69dc3f11a6ba63b10a23b8ca7a7efaf52d5a2d806

Download size:

15.59 GiB

Memory requirement (unknown):

Unknown — No exact 27B hybrid-state/process peak memory evidence for this configuration.

System RAM / disk:

Unknown / Unknown

Placement:

single device. One CPU host; no tensor/pipeline sharding. CPU-only execution; --device none and zero GPU layers. No disk or accelerator offload.

KV cache:

F16 attention K/V cache; Gated DeltaNet recurrent state is separately runtime-managed, with no claimed state precision or full-cache byte estimate.

Workload:

4,096 token context; Unknown; Unknown; concurrency 1; batch 1; cache unknown; repetitions Unknown.

Tokenizer:

Qwen3.5-27B GGUF tokenizer with separately pinned Qwen 27B native Jinja template · 3221f178a6b842d04f1fb42f1c413534adcc0a6a; template fc05daec18b0a78c049392ed2e771dde82bdf654

Sampling / reasoning:

Qwen non-thinking general-task preset: temperature 0.7, top_p 0.8, top_k 20, min_p 0.0, presence_penalty 1.5, repeat_penalty 1.0. Validation max_tokens 512. Non-thinking text-only mode: native Qwen Jinja template, enable_thinking=false, --reasoning off. No /think or /nothink soft switch. Raw output mode; no tool parser, MTP drafter or vision projector.

Runtime prerequisites:

Ubuntu 24.04 x86_64; See prerequisites above.

Official artifact · License

Qwen publishes Qwen3.5-27B under Apache License 2.0; Unsloth’s derivative metadata explicitly names this base model, declares Apache-2.0 and links its license. Read the full license, retain applicable license/copyright/attribution notices and mark modifications when redistributing covered work. Public ungated metadata is not a substitute for license compliance. No provider API subscription is required for the public download.

• qwen35-chat-template.jinja · 0 GiB · revision fc05daec18b0a78c049392ed2e771dde82bdf654

• The full 16,740,812,704-byte GGUF plus attention cache, recurrent state, buffers, OS and setup artifacts need resources. File bytes alone do not prove fit.
• 64 GiB RAM and 80 GiB disk were selected for headroom, not as exact required-memory estimates. No transformer-only KV formula or sibling observations are used.

Install the runtime

1. 1. Build the pinned CPU server

   Install build dependencies and compile the exact official source revision. These packages are system shared dependencies; no CUDA/GPU driver is required. GGML_NATIVE=ON selects -march=native for this GNU/Linux x86 build, compiling for the host CPU. Build and run on the same machine; the binary uses the instruction set exposed by this r7i.2xlarge guest. Do not copy it to a different instance family without rebuilding.

   CommandCopy command

   sudo apt-get update
   sudo apt-get install -y build-essential cmake git curl ca-certificates openssl
   mkdir -p "$HOME/qwen35-27b-hosting/models"
   cd "$HOME/qwen35-27b-hosting"
   git clone https://github.com/ggml-org/llama.cpp.git
   cd llama.cpp
   git checkout --detach 620d2225cc436ab0b001e3200e590e3b286ef9ab
   cmake -S . -B build -DCMAKE_BUILD_TYPE=Release -DGGML_METAL=OFF -DGGML_NATIVE=ON
   cmake --build build --config Release -j 8
   ./build/bin/llama-server --version

   Expected result: Build succeeds and server version identifies pinned source. Stop if checkout or build fails.

Download the model

1. 1. Download the pinned Unsloth GGUF and Qwen template

   Verify both public immutable inputs. Qwen base/template and Unsloth quantization have separate revisions and authors. Use the explicit native Jinja template so non-thinking control does not depend on an uninspected embedded-template version. Stop on either hash mismatch.

   CommandCopy command

   cd "$HOME/qwen35-27b-hosting"
   curl --fail --location --output models/Qwen3.5-27B-Q4_K_M.gguf 'https://huggingface.co/unsloth/Qwen3.5-27B-GGUF/resolve/3221f178a6b842d04f1fb42f1c413534adcc0a6a/Qwen3.5-27B-Q4_K_M.gguf'
   curl --fail --location --output models/qwen35-chat-template.jinja 'https://huggingface.co/Qwen/Qwen3.5-27B/raw/fc05daec18b0a78c049392ed2e771dde82bdf654/chat_template.jinja'
   sha256sum --check <<'SHA256'
   84b5f7f112156d63836a01a69dc3f11a6ba63b10a23b8ca7a7efaf52d5a2d806  models/Qwen3.5-27B-Q4_K_M.gguf
   a4aee8afcf2e0711942cf848899be66016f8d14a889ff9ede07bca099c28f715  models/qwen35-chat-template.jinja
   SHA256

   Expected result: Both files report OK. Exact Q4_K_M GGUF is 16,740,812,704 bytes and native 27B template 7756 bytes. No weights from another size and no projector are substituted.

Configure

1. 1. Create a private service credential

   Generate this only on your own host when following the guide. It is a local service password, not an OpenAI/provider credential. Keep it private and never paste it into tickets. Create it only when absent; preserve an existing key so a repeated setup does not silently rotate access. It is not created during this documentation run.

   CommandCopy command

   cd "$HOME/qwen35-27b-hosting"
   umask 077
   if [ ! -e .api-key ]; then
     openssl rand -hex 32 > .api-key
   fi
   test -s .api-key || { echo 'Existing .api-key is empty; restore it before continuing.'; exit 1; }
   chmod 600 .api-key

   Expected result: A non-empty private .api-key file exists; existing key preserved. No secret is printed.

Start the model

1. 1. Start one CPU-only sequence

   Serve the exact 64-layer Qwen3.5 dense/hybrid 27B GGUF with the separately pinned native 27B template. The pinned qwen35 runtime explicitly recognizes 64 layers as its 27B variant, and CPU dispatch includes Gated DeltaNet and SSM operations. Documented support does not prove this allocation starts or fits. Use enable_thinking=false and --reasoning off; no soft /nothink switch. Keep one sequence, 4,096 total context tokens, and room for 512 output tokens. No mmproj, tools or MTP.

   CommandCopy command

   cd "$HOME/qwen35-27b-hosting"
   export QWEN35_27B_SERVICE_KEY="$(cat .api-key)"
   ./llama.cpp/build/bin/llama-server -m models/Qwen3.5-27B-Q4_K_M.gguf --alias qwen35-27b --host 127.0.0.1 --port 8080 --api-key "$QWEN35_27B_SERVICE_KEY" --device none -ngl 0 -t 8 -c 4096 -np 1 --cache-type-k f16 --cache-type-v f16 --jinja --chat-template-file models/qwen35-chat-template.jinja --chat-template-kwargs '{"enable_thinking":false}' --reasoning off --reasoning-format none --skip-chat-parsing --temp 0.7 --top-p 0.8 --top-k 20 --min-p 0.0 --presence-penalty 1.5 --repeat-penalty 1.0

   Expected result: Model initializes and listener starts. Pass health and authenticated chat checks before relying on the service. Command-line credentials may be visible to privileged/local users; trusted host required.

Verify the setup

1. 1. Check readiness and authenticated text generation

   Use a second server terminal and the private service key. Send only plain text. Check final response against the supplied source passage and verify finish_reason is not length; no exact wording is guaranteed. Do not expect reasoning_content in raw mode. If extended thinking appears, non-thinking setup validation failed.

   CommandCopy command

   cd "$HOME/qwen35-27b-hosting"
   export QWEN35_27B_SERVICE_KEY="$(cat .api-key)"
   curl --fail http://127.0.0.1:8080/health
   curl --fail http://127.0.0.1:8080/v1/chat/completions -H "Authorization: Bearer $QWEN35_27B_SERVICE_KEY" -H 'Content-Type: application/json' -d '{"model":"qwen35-27b","messages":[{"role":"user","content":"Summarize this supplied passage in one sentence without adding facts: The community library opens at 9am on weekdays and closes at 6pm. On Saturday it opens at 10am and closes at 4pm. It is closed on Sunday."}],"max_tokens":512,"temperature":0.7,"top_k":20,"top_p":0.8,"min_p":0.0,"presence_penalty":1.5,"repeat_penalty":1.0}'
   curl -o /dev/null -s -w '%{http_code}\n' http://127.0.0.1:8080/v1/chat/completions -H 'Content-Type: application/json' -d '{"model":"qwen35-27b","messages":[{"role":"user","content":"Hello"}]}'

   Expected result: Health HTTP 200 when ready; authenticated response contains non-empty choices[0].message.content with an accurate final summary and finish_reason not length, without a generated extended thinking trace. Unauthenticated chat returns HTTP 401. Verify facts yourself; output can be wrong.

Performance evidence

No compatible measurement available. Documented support and estimated memory do not establish response speed.

Limits and troubleshooting

• Documented configuration only. No weights, build, inference, full-fit, throughput or latency benchmark was executed by OpenTools.
• Exact Unsloth Q4_K_M artifact is 16,740,812,704 bytes, derived from Qwen/Qwen3.5-27B at separate revisions. Quantization does not guarantee full-precision quality. No weight-size-only memory fit is claimed.
• The native 27B checkpoint is a dense language backbone with a vision encoder, 64 layers and hidden size 5,120. Its config alternates three Gated DeltaNet layers per full-attention layer; it has 16 linear key heads, 48 linear value heads and four full-attention KV heads. Family MoE claims do not make this checkpoint sparse. Hybrid recurrent state and full-attention cache must not be reduced to a transformer-only memory formula.
• This guide is text-only. No mmproj/projector is provided and image/video requests are outside its configuration. The model’s multimodal capability is not a claim that this text recipe validates vision.
• Qwen thinks by default. This guide pins its native template and disables thinking via enable_thinking=false plus --reasoning off. /think and /nothink are not officially supported soft switches. A 4K non-thinking recipe does not preserve/validate full thinking capability; the publisher advises at least 128K context for that purpose.
• The publisher advertises 262,144 native context with extension options. This recipe configures 4,096 total prompt and output, one active sequence and 512 validation output tokens. Longer context, thinking, concurrent users and another cache policy create different configurations.
• Use Qwen’s non-thinking general-task sampler (temperature 0.7, top-p 0.8, top-k 20, min-p 0, presence penalty 1.5 and repetition penalty 1.0). The model card gives conflicting non-thinking reasoning presets in two sections; this recipe avoids that task/mode and uses the consistent general-task preset.
• Summarization validation checks a short text request only. It does not establish coding, multilingual, long-context, retrieval, agentic, structured-output or multimodal quality; generated answers can be wrong.
• Operating system packages and build tool dependencies resolve when installed. Save tool versions/build logs with any actual execution. Cloud hardware is AWS r7i.2xlarge with eight vCPUs and 64 GiB host RAM; eight inference threads are allocated. No exact-instance benchmark is claimed.
• Loopback and a service key do not isolate the service from privileged local users. A shared/public service needs separately designed TLS, authentication and operations.
• The native Transformers config names mamba_ssm_dtype float32, but that does not establish llama.cpp recurrent-state precision or peak memory. This recipe explicitly sets only F16 attention K/V cache; runtime-managed recurrent state remains unmeasured.
• The four-thread, 32 GiB 9B sibling setup is not evidence for this eight-thread, 64 GiB 27B configuration. No throughput, memory or quality observation is transferred from it.

Download or checksum failure

Verify exact uppercase filename and pinned URL; remove only the incomplete dedicated model file and retry. Do not run a mismatching file.

Architecture unsupported or build fails

Confirm exact pinned source commit, Ubuntu compiler/CMake dependencies and GGUF. Do not substitute an older packaged llama.cpp silently.

Out of memory or context overflow

Stop server, free/provision RAM or reduce context and restart; shorten input/output within configured total. Allocation is not a measured guarantee; record any changed configuration.

Connection refused or HTTP 401

Check foreground server logs, 8080 binding, matching private key and Bearer header. Cloud client must have working SSH tunnel to the correct VM; do not expose the port to repair a tunnel.

Unexpected thinking trace, template error or image request failure

Confirm both exact GGUF/template hashes, --jinja, enable_thinking=false and --reasoning off. Do not use /nothink as a substitute. Send only text; this recipe has no mmproj and does not support images/videos. If template or architecture loading fails, confirm exact pinned runtime and stop rather than substituting a different checkpoint.

Stop the model

1. 1. Stop the service

   Press Ctrl+C in server terminal. Unset key in each shell. Close the SSH tunnel with Ctrl+C. Stopping the process alone does not stop EC2 billing.

   CommandCopy command

   unset QWEN35_27B_SERVICE_KEY

   Expected result: Server stops and localhost:8080 stops serving.

Uninstall

1. 1. Remove dedicated model, build and credential files

   Back up wanted logs/files first. Remove only this walkthrough directory after confirming its path. Shared apt packages can remain; do not remove other workloads.

   CommandCopy command

   rm -r -- "$HOME/qwen35-27b-hosting"

   Expected result: Dedicated files and local service credential removed; shared system dependencies remain.

Shut down cloud resources

1. 1. End cloud billing resources

   In EC2 us-east-1, stop for later reuse or terminate this exact VM. Review Delete on termination for its root volume. Delete unwanted detached EBS volumes, snapshots and reserved/Elastic/public IP resources. Stopped VM storage and retained resources may still bill. Check Bills/Cost Explorer later for final usage.

   Expected result: Exact instance is stopped/terminated as intended; no unwanted attached/retained billed resources remain. Instance compute stops only after its lifecycle transition, not when llama-server stops.

Related guides

• https://opentools.ai/llms/qwen35-27b

Exact model and requirements

Artifact:

Unsloth (GGUF quantization); Qwen (base model) / Qwen3.5-27B Q4_K_M, Unsloth derivative · checkpoint unsloth/Qwen3.5-27B-GGUF

Publisher revision:

3221f178a6b842d04f1fb42f1c413534adcc0a6a

Artifact digest:

sha256:84b5f7f112156d63836a01a69dc3f11a6ba63b10a23b8ca7a7efaf52d5a2d806

Download size:

15.59 GiB

Memory requirement (unknown):

Unknown — No exact 27B hybrid-state/process peak memory evidence for this configuration.

System RAM / disk:

Unknown / Unknown

Placement:

single device. One CPU host; no tensor/pipeline sharding. CPU-only execution; --device none and zero GPU layers. No disk or accelerator offload.

KV cache:

F16 attention K/V cache; Gated DeltaNet recurrent state is separately runtime-managed, with no claimed state precision or full-cache byte estimate.

Workload:

4,096 token context; Unknown; Unknown; concurrency 1; batch 1; cache unknown; repetitions Unknown.

Tokenizer:

Qwen3.5-27B GGUF tokenizer with separately pinned Qwen 27B native Jinja template · 3221f178a6b842d04f1fb42f1c413534adcc0a6a; template fc05daec18b0a78c049392ed2e771dde82bdf654

Sampling / reasoning:

Qwen non-thinking general-task preset: temperature 0.7, top_p 0.8, top_k 20, min_p 0.0, presence_penalty 1.5, repeat_penalty 1.0. Validation max_tokens 512. Non-thinking text and one-image chat with exact native 27B template. enable_thinking=false and --reasoning off; raw message.content. CPU language model and CPU projector; no video, tools or MTP.

Runtime prerequisites:

Ubuntu 24.04 x86_64; See prerequisites above.

Official artifact · License

Qwen publishes Qwen3.5-27B under Apache License 2.0; Unsloth’s derivative metadata explicitly names this base model, declares Apache-2.0 and links its license. Read the full license, retain applicable license/copyright/attribution notices and mark modifications when redistributing covered work. Public ungated metadata is not a substitute for license compliance. No provider API subscription is required for the public download.

• qwen35-chat-template.jinja · 0 GiB · revision fc05daec18b0a78c049392ed2e771dde82bdf654
• mmproj-F16.gguf · 0.86 GiB · revision 3221f178a6b842d04f1fb42f1c413534adcc0a6a

• The full 16,740,812,704-byte GGUF plus attention cache, recurrent state, buffers, OS and setup artifacts need resources. File bytes alone do not prove fit.
• 64 GiB RAM and 80 GiB disk were selected for headroom, not as exact required-memory estimates. No transformer-only KV formula or sibling observations are used.
• Image encoder/projector and intermediate image tensors add memory to language-model weights, attention/recurrent state and OS. Projector file size is not its working-memory footprint. Full image configuration fit remains unknown.

Install the runtime

1. 1. Build the pinned CPU server

   Install build dependencies and compile the exact official source revision. These packages are system shared dependencies; no CUDA/GPU driver is required. GGML_NATIVE=ON selects -march=native for this GNU/Linux x86 build, compiling for the host CPU. Build and run on the same machine; no AVX-512 requirement is imposed on the i5-12400. The ordinary llama-server build includes libmtmd image input; Python 3 is only used to create the inert JSON request.

   CommandCopy command

   sudo apt-get update
   sudo apt-get install -y build-essential cmake git curl ca-certificates openssl python3
   mkdir -p "$HOME/qwen35-27b-vision-hosting/models"
   cd "$HOME/qwen35-27b-vision-hosting"
   git clone https://github.com/ggml-org/llama.cpp.git
   cd llama.cpp
   git checkout --detach 620d2225cc436ab0b001e3200e590e3b286ef9ab
   cmake -S . -B build -DCMAKE_BUILD_TYPE=Release -DGGML_METAL=OFF -DGGML_NATIVE=ON
   cmake --build build --config Release -j 8
   ./build/bin/llama-server --version

   Expected result: Build succeeds and server version identifies pinned source. Stop if checkout or build fails.

Download the model

1. 1. Download the pinned Unsloth GGUF and Qwen template

   Verify both public immutable inputs. Qwen base/template and Unsloth quantization have separate revisions and authors. Use the explicit native Jinja template so non-thinking control does not depend on an uninspected embedded-template version. Stop on either hash mismatch.

   CommandCopy command

   cd "$HOME/qwen35-27b-vision-hosting"
   curl --fail --location --output models/Qwen3.5-27B-Q4_K_M.gguf 'https://huggingface.co/unsloth/Qwen3.5-27B-GGUF/resolve/3221f178a6b842d04f1fb42f1c413534adcc0a6a/Qwen3.5-27B-Q4_K_M.gguf'
   curl --fail --location --output models/qwen35-chat-template.jinja 'https://huggingface.co/Qwen/Qwen3.5-27B/raw/fc05daec18b0a78c049392ed2e771dde82bdf654/chat_template.jinja'
   sha256sum --check <<'SHA256'
   84b5f7f112156d63836a01a69dc3f11a6ba63b10a23b8ca7a7efaf52d5a2d806  models/Qwen3.5-27B-Q4_K_M.gguf
   a4aee8afcf2e0711942cf848899be66016f8d14a889ff9ede07bca099c28f715  models/qwen35-chat-template.jinja
   SHA256

   Expected result: Both files report OK. Exact Q4_K_M GGUF is 16,740,812,704 bytes and native 27B template 7756 bytes. No weights from another size and no projector are substituted.

2. 2. Download the exact 27B vision projector

   Use the F16 projector from the same exact Unsloth 27B repository revision. Match its SHA-256; do not replace it with a similarly named sibling file.

   CommandCopy command

   cd "$HOME/qwen35-27b-vision-hosting"
   curl --fail --location --output models/mmproj-F16.gguf 'https://huggingface.co/unsloth/Qwen3.5-27B-GGUF/resolve/3221f178a6b842d04f1fb42f1c413534adcc0a6a/mmproj-F16.gguf'
   printf '%s  %s\n' '458bc46d8f275866fde5d88c9c554d9d462a6e8e3a028090d9850e17ab6a1217' 'models/mmproj-F16.gguf' | sha256sum --check

   Expected result: mmproj-F16.gguf reports OK; metadata size 927,607,040 bytes. Language-model, native-template and projector checks must all pass.

Configure

1. 1. Create a private service credential

   Generate this only on your own host when following the guide. It is a local service password, not an OpenAI/provider credential. Keep it private and never paste it into tickets. Create it only when absent; preserve an existing key so a repeated setup does not silently rotate access. It is not created during this documentation run.

   CommandCopy command

   cd "$HOME/qwen35-27b-vision-hosting"
   umask 077
   if [ ! -e .api-key ]; then
     openssl rand -hex 32 > .api-key
   fi
   test -s .api-key || { echo 'Existing .api-key is empty; restore it before continuing.'; exit 1; }
   chmod 600 .api-key

   Expected result: A non-empty private .api-key file exists; existing key preserved. No secret is printed.

Start the model

1. 1. Start one CPU-only sequence

   Start the exact 27B language model plus matching projector on CPU. --no-mmproj-offload and --mmproj-device none disable projector GPU offload; the text model also uses --device none and zero GPU layers. Native template disables thinking. Set one active sequence, 4,096 total context and a nominal 1,024-token minimum/maximum per image, with image encoding batches limited to 1,024 tokens. This is a preprocessing/token-budget setting, not a measured exact image token count or per-image fit guarantee. Watch model/projector initialization and verify both text and image requests below.

   CommandCopy command

   cd "$HOME/qwen35-27b-vision-hosting"
   export QWEN35_27B_VISION_SERVICE_KEY="$(cat .api-key)"
   ./llama.cpp/build/bin/llama-server -m models/Qwen3.5-27B-Q4_K_M.gguf --alias qwen35-27b --host 127.0.0.1 --port 8080 --api-key "$QWEN35_27B_VISION_SERVICE_KEY" --device none -ngl 0 -t 8 -c 4096 -np 1 --cache-type-k f16 --cache-type-v f16 --jinja --chat-template-file models/qwen35-chat-template.jinja --chat-template-kwargs '{"enable_thinking":false}' --reasoning off --reasoning-format none --skip-chat-parsing --temp 0.7 --top-p 0.8 --top-k 20 --min-p 0.0 --presence-penalty 1.5 --repeat-penalty 1.0 --mmproj models/mmproj-F16.gguf --no-mmproj-offload --mmproj-device none --image-min-tokens 1024 --image-max-tokens 1024 --mtmd-batch-max-tokens 1024

   Expected result: Model initializes and listener starts. Pass health and authenticated chat checks before relying on the service. Command-line credentials may be visible to privileged/local users; trusted host required.

Verify the setup

1. 1. Check readiness and authenticated text generation

   Use a second server terminal and the private service key. Send only plain text. Check final response against the supplied source passage and verify finish_reason is not length; no exact wording is guaranteed. Do not expect reasoning_content in raw mode. If extended thinking appears, non-thinking setup validation failed.

   CommandCopy command

   cd "$HOME/qwen35-27b-vision-hosting"
   export QWEN35_27B_VISION_SERVICE_KEY="$(cat .api-key)"
   curl --fail http://127.0.0.1:8080/health
   curl --fail http://127.0.0.1:8080/v1/chat/completions -H "Authorization: Bearer $QWEN35_27B_VISION_SERVICE_KEY" -H 'Content-Type: application/json' -d '{"model":"qwen35-27b","messages":[{"role":"user","content":"Summarize this supplied passage in one sentence without adding facts: The community library opens at 9am on weekdays and closes at 6pm. On Saturday it opens at 10am and closes at 4pm. It is closed on Sunday."}],"max_tokens":512,"temperature":0.7,"top_k":20,"top_p":0.8,"min_p":0.0,"presence_penalty":1.5,"repeat_penalty":1.0}'
   curl -o /dev/null -s -w '%{http_code}\n' http://127.0.0.1:8080/v1/chat/completions -H 'Content-Type: application/json' -d '{"model":"qwen35-27b","messages":[{"role":"user","content":"Hello"}]}'

   Expected result: Health HTTP 200 when ready; authenticated response contains non-empty choices[0].message.content with an accurate final summary and finish_reason not length, without a generated extended thinking trace. Unauthenticated chat returns HTTP 401. Verify facts yourself; output can be wrong.

2. 2. Check one image without uploading it to a third-party inference service

   Download the immutable llama.cpp JPEG fixture and verify its bytes. It is a New York Times front page; the largest headline is MEN WALK ON MOON. Build an authenticated image_url data-URI request on the server. The image is sent to this private local endpoint. One image, a short prompt and a 512-token output cap are the walkthrough workload. Actual embedding tokens must be checked in runtime logs/usage; no exact token or latency value is assumed.

   CommandCopy command

   cd "$HOME/qwen35-27b-vision-hosting"
   curl --fail --location --output fixture.jpg 'https://raw.githubusercontent.com/ggml-org/llama.cpp/620d2225cc436ab0b001e3200e590e3b286ef9ab/tools/mtmd/test-1.jpeg'
   printf '%s  %s\n' '2dff664c0c8aaea18aff8cbe7e868845b775e90cdd7a0bac98df709b131deaa3' 'fixture.jpg' | sha256sum --check
   python3 - <<'PYREQUEST'
   import base64, json, pathlib
   image = base64.b64encode(pathlib.Path('fixture.jpg').read_bytes()).decode()
   request = {'model':'qwen35-27b','messages':[{'role':'user','content':[{'type':'text','text':'Identify the newspaper and transcribe its largest headline. Then briefly describe the front page.'},{'type':'image_url','image_url':{'url':'data:image/jpeg;base64,' + image}}]}],'max_tokens':512,'temperature':0.7,'top_p':0.8,'top_k':20,'min_p':0.0,'presence_penalty':1.5,'repeat_penalty':1.0}
   pathlib.Path('image-request.json').write_text(json.dumps(request))
   PYREQUEST
   export QWEN35_27B_VISION_SERVICE_KEY="$(cat .api-key)"
   curl --fail http://127.0.0.1:8080/v1/chat/completions -H "Authorization: Bearer $QWEN35_27B_VISION_SERVICE_KEY" -H 'Content-Type: application/json' --data-binary @image-request.json

   Expected result: Fixture checksum OK. Response choices[0].message.content identifies The New York Times and the largest headline MEN WALK ON MOON, with a completed final answer and finish_reason not length. Inspect against the actual image; no exact wording is guaranteed. Missing image understanding, wrong headline, context error or truncated output fails this functional check. This expected result is a visual fixture check, not an executed model result.

Performance evidence

No compatible measurement available. Documented support and estimated memory do not establish response speed.

Limits and troubleshooting

• Documented configuration only. No weights, build, inference, full-fit, throughput or latency benchmark was executed by OpenTools.
• Exact Unsloth Q4_K_M artifact is 16,740,812,704 bytes, derived from Qwen/Qwen3.5-27B at separate revisions. Quantization does not guarantee full-precision quality. No weight-size-only memory fit is claimed.
• The native 27B checkpoint is a dense language backbone with a vision encoder, 64 layers and hidden size 5,120. Its config alternates three Gated DeltaNet layers per full-attention layer; it has 16 linear key heads, 48 linear value heads and four full-attention KV heads. Family MoE claims do not make this checkpoint sparse. Hybrid recurrent state and full-attention cache must not be reduced to a transformer-only memory formula.
• Qwen thinks by default. This guide pins its native template and disables thinking via enable_thinking=false plus --reasoning off. /think and /nothink are not officially supported soft switches. A 4K non-thinking recipe does not preserve/validate full thinking capability; the publisher advises at least 128K context for that purpose.
• The publisher advertises 262,144 native context with extension options. This recipe configures 4,096 total prompt and output, one active sequence and 512 validation output tokens. Longer context, thinking, concurrent users and another cache policy create different configurations.
• Use Qwen’s non-thinking general-task sampler (temperature 0.7, top-p 0.8, top-k 20, min-p 0, presence penalty 1.5 and repetition penalty 1.0). The model card gives conflicting non-thinking reasoning presets in two sections; this recipe avoids that task/mode and uses the consistent general-task preset.
• Operating system packages and build tool dependencies resolve when installed. Save tool versions/build logs with any actual execution. Local hardware is Intel Core i5-12400 with eight inference threads allocated; no exact-hardware benchmark is claimed.
• Loopback and a service key do not isolate the service from privileged local users. A shared/public service needs separately designed TLS, authentication and operations.
• The native Transformers config names mamba_ssm_dtype float32, but that does not establish llama.cpp recurrent-state precision or peak memory. This recipe explicitly sets only F16 attention K/V cache; runtime-managed recurrent state remains unmeasured.
• The four-thread, 32 GiB 9B sibling setup is not evidence for this eight-thread, 64 GiB 27B configuration. No throughput, memory or quality observation is transferred from it.
• This item supports documented text plus one-image input, not video, audio, a multi-image workload or an OCR quality guarantee. The newspaper headline check is narrow and should be manually inspected.
• The native 27B model class Qwen3_5ForConditionalGeneration is explicitly registered by the pinned projector converter, which emits QWEN3VL. Its 27B vision_config has patch size 16, merge size 2 and projection output size 5,120. Pinned mtmd/clip source handles QWEN3VL preprocessing. These sources establish documented mapping, not an executed projector-compatibility result.
• The configured image minimum/maximum of 1,024 tokens bounds intended dynamic-resolution preprocessing; actual rounded image/token/template overhead and full process memory remain unmeasured. A 4,096-token total context is not a guarantee for arbitrary pictures, multiple images, long prompts or output. Reject context errors; keep this one-image short-prompt workload.
• The pinned runtime warns that grounding tasks need at least 1,024 image tokens; this guide uses that nominal budget, but does not validate grounding or fine-print OCR accuracy. Changing resolution/image budget changes the configuration.
• The fixture was independently visually inspected by the guide author, but no inference was run. Its expected headline is not a model measurement. Do not transfer 9B projector compatibility, performance or quality.

Download or checksum failure

Verify exact uppercase filename and pinned URL; remove only the incomplete dedicated model file and retry. Do not run a mismatching file.

Architecture unsupported or build fails

Confirm exact pinned source commit, Ubuntu compiler/CMake dependencies and GGUF. Do not substitute an older packaged llama.cpp silently.

Out of memory or context overflow

Stop server, free/provision RAM or reduce context and restart; shorten input/output within configured total. Allocation is not a measured guarantee; record any changed configuration.

Connection refused or HTTP 401

Check foreground server logs, 8080 binding, matching private key and Bearer header. Run the client on this same desktop and keep the loopback listener.

Projector mismatch, image ignored, wrong headline or context overflow

Check all three exact hashes, --mmproj path, CPU projector flags, native 27B template and content-array image_url with a JPEG base64 data URI. Do not send the file path as plain text. Check runtime logs for actual image-token/context use. Retry only the supplied one-image short prompt; reject wrong/truncated output. More image tokens or a larger context is a changed configuration and needs adequate resources; do not infer fit.

Stop the model

1. 1. Stop the service

   Press Ctrl+C in server terminal. Unset key in each shell. Confirm this desktop localhost:8080 no longer serves the model.

   CommandCopy command

   unset QWEN35_27B_VISION_SERVICE_KEY

   Expected result: Server stops and localhost:8080 stops serving.

Uninstall

1. 1. Remove dedicated model, build and credential files

   Back up wanted logs/files first. Remove only this walkthrough directory after confirming its path. Shared apt packages can remain; do not remove other workloads.

   CommandCopy command

   rm -r -- "$HOME/qwen35-27b-vision-hosting"

   Expected result: Dedicated files and local service credential removed; shared system dependencies remain.

Related guides

• https://opentools.ai/llms/qwen35-27b

Exact model and requirements

Artifact:

Unsloth (GGUF quantization); Qwen (base model) / Qwen3.5-27B Q4_K_M, Unsloth derivative · checkpoint unsloth/Qwen3.5-27B-GGUF

Publisher revision:

3221f178a6b842d04f1fb42f1c413534adcc0a6a

Artifact digest:

sha256:84b5f7f112156d63836a01a69dc3f11a6ba63b10a23b8ca7a7efaf52d5a2d806

Download size:

15.59 GiB

Memory requirement (unknown):

Unknown — No exact 27B hybrid-state/process peak memory evidence for this configuration.

System RAM / disk:

Unknown / Unknown

Placement:

single device. One CPU host; no tensor/pipeline sharding. CPU-only execution; --device none and zero GPU layers. No disk or accelerator offload.

KV cache:

F16 attention K/V cache; Gated DeltaNet recurrent state is separately runtime-managed, with no claimed state precision or full-cache byte estimate.

Workload:

8,192 token context; Unknown; Unknown; concurrency 1; batch 1; cache unknown; repetitions Unknown.

Tokenizer:

Qwen3.5-27B GGUF tokenizer with separately pinned Qwen 27B native Jinja template · 3221f178a6b842d04f1fb42f1c413534adcc0a6a; template fc05daec18b0a78c049392ed2e771dde82bdf654

Sampling / reasoning:

Qwen non-thinking general-task preset: temperature 0.7, top_p 0.8, top_k 20, min_p 0.0, presence_penalty 1.5, repeat_penalty 1.0. Validation max_tokens 512. Non-thinking short-video general understanding; the exact 27B native Jinja template enable_thinking=false and --reasoning off, consistent publisher general-task sampler. Native input_video with whole immutable 10.03-second fixture, target 0.5 fps and 1,000 ms timestamps; no audio processing. Temporal readout is generated final content, not exact predicted wording.

Runtime prerequisites:

Ubuntu 24.04 x86_64; See prerequisites above.

Official artifact · License

Qwen publishes Qwen3.5-27B under Apache License 2.0; Unsloth’s derivative metadata explicitly names this base model, declares Apache-2.0 and links its license. Read the full license, retain applicable license/copyright/attribution notices and mark modifications when redistributing covered work. Public ungated metadata is not a substitute for license compliance. No provider API subscription is required for the public download.

• qwen35-chat-template.jinja · 0 GiB · revision fc05daec18b0a78c049392ed2e771dde82bdf654
• mmproj-F16.gguf · 0.86 GiB · revision 3221f178a6b842d04f1fb42f1c413534adcc0a6a

• The full 16,740,812,704-byte GGUF plus attention cache, recurrent state, buffers, OS and setup artifacts need resources. File bytes alone do not prove fit.
• 64 GiB RAM and 80 GiB disk were selected for headroom, not as exact required-memory estimates. No transformer-only KV formula or sibling observations are used.
• Image encoder/projector and intermediate image tensors add memory to language-model weights, attention/recurrent state and OS. Projector file size is not its working-memory footprint. Full video configuration fit remains unknown.
• The 8,192-token video context differs from the 4K image guide. FFmpeg/ffprobe processes, decoded frames, frame embeddings, warmup and larger hybrid cache/state add unmeasured RAM. 64 GiB allocation remains a planning choice, not full-fit proof.

Install the runtime

1. 1. Build the pinned CPU server

   Install build tools/Python, build FFmpeg n8.0.1 from its exact source commit into this walkthrough directory, then compile the exact llama.cpp CPU server with video and subprocess support. FFmpeg automatic external-library detection and network input are disabled for this local base64-file workflow; built-in H.264 decoding remains enabled. No new packages are installed during this research; these are reader commands. Preserve version/build logs.

   CommandCopy command

   sudo apt-get update
   sudo apt-get install -y build-essential cmake git curl ca-certificates openssl pkg-config python3
   mkdir -p "$HOME/qwen35-27b-video-hosting/models"
   cd "$HOME/qwen35-27b-video-hosting"
   git clone https://github.com/FFmpeg/FFmpeg.git ffmpeg-source
   git -C ffmpeg-source checkout --detach 894da5ca7d742e4429ffb2af534fcda0103ef593
   cd ffmpeg-source
   ./configure --prefix="$HOME/qwen35-27b-video-hosting/ffmpeg-bin" --disable-x86asm --disable-doc --disable-debug --disable-autodetect --disable-network
   make -j 8
   make install
   cd ..
   export PATH="$HOME/qwen35-27b-video-hosting/ffmpeg-bin/bin:$PATH"
   ffmpeg -version
   ffprobe -version
   git clone https://github.com/ggml-org/llama.cpp.git
   cd llama.cpp
   git checkout --detach 620d2225cc436ab0b001e3200e590e3b286ef9ab
   cmake -S . -B build -DCMAKE_BUILD_TYPE=Release -DGGML_METAL=OFF -DGGML_NATIVE=ON -DMTMD_VIDEO=ON -DLLAMA_SUBPROCESS=ON
   cmake --build build --config Release -j 8
   ./build/bin/llama-server --version

   Expected result: Pinned ffmpeg and ffprobe binaries report n8.0.1/source revision; llama-server build succeeds with MTMD_VIDEO and LLAMA_SUBPROCESS enabled. No model inference has run yet.

Download the model

1. 1. Download the pinned Unsloth GGUF and Qwen template

   Verify both public immutable inputs. Qwen base/template and Unsloth quantization have separate revisions and authors. Use the explicit native Jinja template so non-thinking control does not depend on an uninspected embedded-template version. Stop on either hash mismatch.

   CommandCopy command

   cd "$HOME/qwen35-27b-video-hosting"
   curl --fail --location --output models/Qwen3.5-27B-Q4_K_M.gguf 'https://huggingface.co/unsloth/Qwen3.5-27B-GGUF/resolve/3221f178a6b842d04f1fb42f1c413534adcc0a6a/Qwen3.5-27B-Q4_K_M.gguf'
   curl --fail --location --output models/qwen35-chat-template.jinja 'https://huggingface.co/Qwen/Qwen3.5-27B/raw/fc05daec18b0a78c049392ed2e771dde82bdf654/chat_template.jinja'
   sha256sum --check <<'SHA256'
   84b5f7f112156d63836a01a69dc3f11a6ba63b10a23b8ca7a7efaf52d5a2d806  models/Qwen3.5-27B-Q4_K_M.gguf
   a4aee8afcf2e0711942cf848899be66016f8d14a889ff9ede07bca099c28f715  models/qwen35-chat-template.jinja
   SHA256

   Expected result: Both files report OK. Exact Q4_K_M GGUF is 16,740,812,704 bytes and native 27B template 7756 bytes. No weights from another size and no projector are substituted.

2. 2. Download the exact 27B vision projector

   Use the F16 projector from the same exact Unsloth 27B repository revision. Match its SHA-256; do not replace it with a similarly named sibling file.

   CommandCopy command

   cd "$HOME/qwen35-27b-video-hosting"
   curl --fail --location --output models/mmproj-F16.gguf 'https://huggingface.co/unsloth/Qwen3.5-27B-GGUF/resolve/3221f178a6b842d04f1fb42f1c413534adcc0a6a/mmproj-F16.gguf'
   printf '%s  %s\n' '458bc46d8f275866fde5d88c9c554d9d462a6e8e3a028090d9850e17ab6a1217' 'models/mmproj-F16.gguf' | sha256sum --check

   Expected result: mmproj-F16.gguf reports OK; metadata size 927,607,040 bytes. Language-model, native-template and projector checks must all pass.

Configure

1. 1. Create a private service credential

   Generate this only on your own host when following the guide. It is a local service password, not an OpenAI/provider credential. Keep it private and never paste it into tickets. Create it only when absent; preserve an existing key so a repeated setup does not silently rotate access. It is not created during this documentation run.

   CommandCopy command

   cd "$HOME/qwen35-27b-video-hosting"
   umask 077
   if [ ! -e .api-key ]; then
     openssl rand -hex 32 > .api-key
   fi
   test -s .api-key || { echo 'Existing .api-key is empty; restore it before continuing.'; exit 1; }
   chmod 600 .api-key

   Expected result: A non-empty private .api-key file exists; existing key preserved. No secret is printed.

Start the model

1. 1. Start one CPU-only sequence

   Use the same exact 27B language/projector/template pins with explicitly video-enabled build. Server helper uses FFmpeg fps filter at 0.5 fps across the entire input and adds frame timestamps every 1,000 ms; with 2-second sample spacing each frame gets a timestamp. QWEN3VL may merge adjacent frame pairs; do not assume reduction. Conservative five-frame budget at 1,024 tokens each plus 512 output tokens is 5,632; 8,192 leaves 2,560 tokens for prompt/template/timestamps. Actual counts unmeasured. Multimodal runtime disables context shifting; stop on overflow rather than silently change configuration.

   CommandCopy command

   cd "$HOME/qwen35-27b-video-hosting"
   export QWEN35_27B_VIDEO_SERVICE_KEY="$(cat .api-key)"
   ./llama.cpp/build/bin/llama-server -m models/Qwen3.5-27B-Q4_K_M.gguf --alias qwen35-27b --host 127.0.0.1 --port 8080 --api-key "$QWEN35_27B_VIDEO_SERVICE_KEY" --device none -ngl 0 -t 8 -c 8192 -np 1 --cache-type-k f16 --cache-type-v f16 --jinja --chat-template-file models/qwen35-chat-template.jinja --chat-template-kwargs '{"enable_thinking":false}' --reasoning off --reasoning-format none --skip-chat-parsing --temp 0.7 --top-p 0.8 --top-k 20 --min-p 0.0 --presence-penalty 1.5 --repeat-penalty 1.0 --mmproj models/mmproj-F16.gguf --no-mmproj-offload --mmproj-device none --image-min-tokens 1024 --image-max-tokens 1024 --mtmd-batch-max-tokens 1024 --video-fps 0.5 --video-timestamp-interval 1000 --video-ffmpeg-dir "$HOME/qwen35-27b-video-hosting/ffmpeg-bin/bin"

   Expected result: Model initializes and listener starts. Pass health and authenticated chat checks before relying on the service. Command-line credentials may be visible to privileged/local users; trusted host required.

Verify the setup

1. 1. Check readiness and authenticated text generation

   Use a second server terminal and the private service key. Send only plain text. Check final response against the supplied source passage and verify finish_reason is not length; no exact wording is guaranteed. Do not expect reasoning_content in raw mode. If extended thinking appears, non-thinking setup validation failed.

   CommandCopy command

   cd "$HOME/qwen35-27b-video-hosting"
   export QWEN35_27B_VIDEO_SERVICE_KEY="$(cat .api-key)"
   curl --fail http://127.0.0.1:8080/health
   curl --fail http://127.0.0.1:8080/v1/chat/completions -H "Authorization: Bearer $QWEN35_27B_VIDEO_SERVICE_KEY" -H 'Content-Type: application/json' -d '{"model":"qwen35-27b","messages":[{"role":"user","content":"Summarize this supplied passage in one sentence without adding facts: The community library opens at 9am on weekdays and closes at 6pm. On Saturday it opens at 10am and closes at 4pm. It is closed on Sunday."}],"max_tokens":512,"temperature":0.7,"top_k":20,"top_p":0.8,"min_p":0.0,"presence_penalty":1.5,"repeat_penalty":1.0}'
   curl -o /dev/null -s -w '%{http_code}\n' http://127.0.0.1:8080/v1/chat/completions -H 'Content-Type: application/json' -d '{"model":"qwen35-27b","messages":[{"role":"user","content":"Hello"}]}'

   Expected result: Health HTTP 200 when ready; authenticated response contains non-empty choices[0].message.content with an accurate final summary and finish_reason not length, without a generated extended thinking trace. Unauthenticated chat returns HTTP 401. Verify facts yourself; output can be wrong.

2. 2. Verify all sampled frames before video submission

   Download and hash-check the whole immutable moving MP4. ffprobe verifies source metadata. Decode the entire file through the same fixed fps=0.500000 RGB24 filter as the native helper; count all bytes, with no -frames or artificial stop. Reject a changed source, oversized duration, partial frame or any count other than five. This is media decoding only, not model inference.

   CommandCopy command

   cd "$HOME/qwen35-27b-video-hosting"
   export PATH="$HOME/qwen35-27b-video-hosting/ffmpeg-bin/bin:$PATH"
   curl --fail --location --output fixture.mp4 'https://raw.githubusercontent.com/ggml-org/llama.cpp/620d2225cc436ab0b001e3200e590e3b286ef9ab/tools/mtmd/test-3.mp4'
   printf '%s  %s\n' '95a9776a0e7f8d6ab1ba67a31dbb09a4c04ffda3c853fd933624e3496fa970a5' 'fixture.mp4' | sha256sum --check
   ffprobe -v error -select_streams v:0 -show_entries stream=width,height,codec_name:format=duration -of json fixture.mp4 > fixture-metadata.json
   ffmpeg -y -v error -i fixture.mp4 -vf fps=0.500000 -an -f rawvideo -pix_fmt rgb24 fixture-rgb.raw
   python3 - <<'PYPREFLIGHT'
   import json,pathlib
   m=json.loads(pathlib.Path('fixture-metadata.json').read_text())
   s=m['streams'][0]
   assert s['width']==854 and s['height']==358 and s['codec_name']=='h264'
   assert 10.02 <= float(m['format']['duration']) <= 10.04 and float(m['format']['duration']) <= 10.1
   size=pathlib.Path('fixture-rgb.raw').stat().st_size
   frame_bytes=854*358*3
   assert size % frame_bytes == 0, 'partial decoded frame'
   assert size // frame_bytes == 5, 'reject: expected exactly five sampled frames'
   print('Preflight OK: complete five-frame bounded input')
   PYPREFLIGHT

   Expected result: Fixture SHA OK; metadata matches; preflight reports exactly five complete sampled frames. If any assertion fails, stop before video submission. No generated frame limit hides additional decoded frames.

3. 3. Verify grounded temporal video understanding

   Run from the server host or its SSH shell. Activate the same tools by explicit paths. Native request uses input_video.data raw base64, not an image_url still. Inspect the response for temporal order: a man is initially seated under a white cape beside a large gray character; a hand reaches or pulls a chair-side handle; later the man is out of the chair, standing or moving around the room. A static description alone fails. Exact dialogue, names, action timing and prose are not pass criteria; video path does not evaluate audio. Confirm finish_reason is not length and no extended thinking trace. Before the video request, authenticated /props must report modalities.vision=true and modalities.video=true. A successful image load alone is insufficient.

   CommandCopy command

   cd "$HOME/qwen35-27b-video-hosting"
   export QWEN35_27B_VIDEO_SERVICE_KEY="$(cat .api-key)"
   curl --fail http://127.0.0.1:8080/props -H "Authorization: Bearer $QWEN35_27B_VIDEO_SERVICE_KEY" --output video-props.json
   python3 - <<'PYPROPS'
   import json
   p=json.load(open('video-props.json'))
   assert p['modalities']['vision'] and p['modalities']['video'], 'Loaded runtime does not report video capability'
   PYPROPS
   python3 - <<'PYREQUEST'
   import base64,json,pathlib
   video=base64.b64encode(pathlib.Path('fixture.mp4').read_bytes()).decode()
   request={'model':'qwen35-27b','messages':[{'role':'user','content':[{'type':'text','text':'Describe the main physical action in this video in order. Explain how the man and chair change between the beginning, middle and end. Use only visible evidence; do not invent dialogue.'},{'type':'input_video','input_video':{'data':video}}]}],'max_tokens':512,'temperature':0.7,'top_p':0.8,'top_k':20,'min_p':0.0,'presence_penalty':1.5,'repeat_penalty':1.0}
   pathlib.Path('video-request.json').write_text(json.dumps(request))
   PYREQUEST
   curl --fail http://127.0.0.1:8080/v1/chat/completions -H "Authorization: Bearer $QWEN35_27B_VIDEO_SERVICE_KEY" -H 'Content-Type: application/json' --data-binary @video-request.json --output video-response.json
   cat video-response.json

   Expected result: Authenticated HTTP 200 with nonempty choices[0].message.content describing the visible before/after action in correct order, finish_reason not length, no extended thinking. Unsupported-video errors, truncation, incorrect order or invented actions fail. No exact generated response or confidence promised.

Performance evidence

No compatible measurement available. Documented support and estimated memory do not establish response speed.

Limits and troubleshooting

• Documented configuration only. No weights, build, inference, full-fit, throughput or latency benchmark was executed by OpenTools.
• Exact Unsloth Q4_K_M artifact is 16,740,812,704 bytes, derived from Qwen/Qwen3.5-27B at separate revisions. Quantization does not guarantee full-precision quality. No weight-size-only memory fit is claimed.
• The native 27B checkpoint is a dense language backbone with a vision encoder, 64 layers and hidden size 5,120. Its config alternates three Gated DeltaNet layers per full-attention layer; it has 16 linear key heads, 48 linear value heads and four full-attention KV heads. Family MoE claims do not make this checkpoint sparse. Hybrid recurrent state and full-attention cache must not be reduced to a transformer-only memory formula.
• Use Qwen’s non-thinking general-task sampler (temperature 0.7, top-p 0.8, top-k 20, min-p 0, presence penalty 1.5 and repetition penalty 1.0). The model card gives conflicting non-thinking reasoning presets in two sections; this recipe avoids that task/mode and uses the consistent general-task preset.
• Operating system packages and build tool dependencies resolve when installed. Save tool versions/build logs with any actual execution. Local hardware is Intel Core i5-12400 with eight inference threads allocated; no exact-hardware benchmark is claimed.
• Loopback and a service key do not isolate the service from privileged local users. A shared/public service needs separately designed TLS, authentication and operations.
• The native Transformers config names mamba_ssm_dtype float32, but that does not establish llama.cpp recurrent-state precision or peak memory. This recipe explicitly sets only F16 attention K/V cache; runtime-managed recurrent state remains unmeasured.
• The four-thread, 32 GiB 9B sibling setup is not evidence for this eight-thread, 64 GiB 27B configuration. No throughput, memory or quality observation is transferred from it.
• The native 27B model class Qwen3_5ForConditionalGeneration is explicitly registered by the pinned projector converter, which emits QWEN3VL. Its 27B vision_config has patch size 16, merge size 2 and projection output size 5,120. Pinned mtmd/clip source handles QWEN3VL preprocessing. These sources establish documented mapping, not an executed projector-compatibility result.
• The pinned runtime warns that grounding tasks need at least 1,024 image tokens; this guide uses that nominal budget, but does not validate grounding or fine-print OCR accuracy. Changing resolution/image budget changes the configuration.
• Source fixture duration is about 10.03 seconds. Target 0.5 fps plans five sampled frames; actual decoded frame count is a preflight gate, not a claimed model observation. Native helper has no input duration/frame stop parameter and processes the whole clip. Do not substitute longer videos.
• Temporal frame grouping, image resizing/token bounds, timestamps and template all consume context. 8,192 is an author configuration with conservative reserve, not the full advertised model context. Actual prompt tokens and peak RAM unknown. No long-video/audio benchmark or still-image quality result transfers.
• FFmpeg fps sampling can select source frames near interval midpoints, rather than exact 0, 2, 4, 6 or 8-second seeks. Server timestamps are nominal sampled-frame index divided by target fps, not verified original-source event times. Judge visible action ordering, not exact action timestamps. Video audio is not processed by this configuration.
• Thinking is disabled for this bounded general-task recipe. The 27B publisher recommends at least 128K context to preserve thinking capabilities; this 8K non-thinking setup does not validate those capabilities or long-video understanding.

Download or checksum failure

Verify exact uppercase filename and pinned URL; remove only the incomplete dedicated model file and retry. Do not run a mismatching file.

Architecture unsupported or build fails

Confirm exact pinned source commit, Ubuntu compiler/CMake dependencies and GGUF. Do not substitute an older packaged llama.cpp silently.

Out of memory or context overflow

Stop server, free/provision RAM or reduce context and restart; shorten input/output within configured total. Allocation is not a measured guarantee; record any changed configuration.

Connection refused or HTTP 401

Check foreground server logs, 8080 binding, matching private key and Bearer header. Run the client on this same desktop and keep the loopback listener.

Unsupported video or incorrect temporal answer

Confirm matching 27B projector, MTMD_VIDEO=ON and LLAMA_SUBPROCESS=ON, exact FFmpeg tools, input_video.data schema and frame-count preflight. Confirm --video-fps 0.5 and 8,192 context tokens. If logs show unavailable video, overflow or wrong action order stop; do not treat still-image fallback as video support.

Stop the model

1. 1. Stop the service

   Press Ctrl+C in server terminal. Unset key in each shell. Confirm this desktop localhost:8080 no longer serves the model.

   CommandCopy command

   unset QWEN35_27B_VIDEO_SERVICE_KEY

   Expected result: Server stops and localhost:8080 stops serving.

Uninstall

1. 1. Remove dedicated model, build and credential files

   Back up wanted logs/files first. Remove only this walkthrough directory after confirming its path. Shared apt packages can remain; do not remove other workloads.

   CommandCopy command

   rm -r -- "$HOME/qwen35-27b-video-hosting"

   Expected result: Dedicated files and local service credential removed; shared system dependencies remain.

Related guides

• https://opentools.ai/llms/qwen35-27b

Exact model and requirements

Artifact:

Unsloth (GGUF quantization); Qwen (base model) / Qwen3.5-27B Q4_K_M, Unsloth derivative · checkpoint unsloth/Qwen3.5-27B-GGUF

Publisher revision:

3221f178a6b842d04f1fb42f1c413534adcc0a6a

Artifact digest:

sha256:84b5f7f112156d63836a01a69dc3f11a6ba63b10a23b8ca7a7efaf52d5a2d806

Download size:

15.59 GiB

Memory requirement (unknown):

Unknown — No exact 27B hybrid-state/process peak memory evidence for this configuration.

System RAM / disk:

Unknown / Unknown

Placement:

single device. One CPU host; no tensor/pipeline sharding. CPU-only execution; --device none and zero GPU layers. No disk or accelerator offload.

KV cache:

F16 attention K/V cache; Gated DeltaNet recurrent state is separately runtime-managed, with no claimed state precision or full-cache byte estimate.

Workload:

8,192 token context; Unknown; Unknown; concurrency 1; batch 1; cache unknown; repetitions Unknown.

Tokenizer:

Qwen3.5-27B GGUF tokenizer with separately pinned Qwen 27B native Jinja template · 3221f178a6b842d04f1fb42f1c413534adcc0a6a; template fc05daec18b0a78c049392ed2e771dde82bdf654

Sampling / reasoning:

Qwen non-thinking general-task preset: temperature 0.7, top_p 0.8, top_k 20, min_p 0.0, presence_penalty 1.5, repeat_penalty 1.0. Validation max_tokens 512. Non-thinking short-video general understanding; the exact 27B native Jinja template enable_thinking=false and --reasoning off, consistent publisher general-task sampler. Native input_video with whole immutable 10.03-second fixture, target 0.5 fps and 1,000 ms timestamps; no audio processing. Temporal readout is generated final content, not exact predicted wording.

Runtime prerequisites:

Ubuntu 24.04 x86_64; See prerequisites above.

Official artifact · License

Qwen publishes Qwen3.5-27B under Apache License 2.0; Unsloth’s derivative metadata explicitly names this base model, declares Apache-2.0 and links its license. Read the full license, retain applicable license/copyright/attribution notices and mark modifications when redistributing covered work. Public ungated metadata is not a substitute for license compliance. No provider API subscription is required for the public download.

• qwen35-chat-template.jinja · 0 GiB · revision fc05daec18b0a78c049392ed2e771dde82bdf654
• mmproj-F16.gguf · 0.86 GiB · revision 3221f178a6b842d04f1fb42f1c413534adcc0a6a

• The full 16,740,812,704-byte GGUF plus attention cache, recurrent state, buffers, OS and setup artifacts need resources. File bytes alone do not prove fit.
• 64 GiB RAM and 80 GiB disk were selected for headroom, not as exact required-memory estimates. No transformer-only KV formula or sibling observations are used.
• Image encoder/projector and intermediate image tensors add memory to language-model weights, attention/recurrent state and OS. Projector file size is not its working-memory footprint. Full video configuration fit remains unknown.
• The 8,192-token video context differs from the 4K image guide. FFmpeg/ffprobe processes, decoded frames, frame embeddings, warmup and larger hybrid cache/state add unmeasured RAM. 64 GiB allocation remains a planning choice, not full-fit proof.

Install the runtime

1. 1. Build the pinned CPU server

   Install build tools/Python, build FFmpeg n8.0.1 from its exact source commit into this walkthrough directory, then compile the exact llama.cpp CPU server with video and subprocess support. FFmpeg automatic external-library detection and network input are disabled for this local base64-file workflow; built-in H.264 decoding remains enabled. No new packages are installed during this research; these are reader commands. Preserve version/build logs.

   CommandCopy command

   sudo apt-get update
   sudo apt-get install -y build-essential cmake git curl ca-certificates openssl pkg-config python3
   mkdir -p "$HOME/qwen35-27b-video-hosting/models"
   cd "$HOME/qwen35-27b-video-hosting"
   git clone https://github.com/FFmpeg/FFmpeg.git ffmpeg-source
   git -C ffmpeg-source checkout --detach 894da5ca7d742e4429ffb2af534fcda0103ef593
   cd ffmpeg-source
   ./configure --prefix="$HOME/qwen35-27b-video-hosting/ffmpeg-bin" --disable-x86asm --disable-doc --disable-debug --disable-autodetect --disable-network
   make -j 8
   make install
   cd ..
   export PATH="$HOME/qwen35-27b-video-hosting/ffmpeg-bin/bin:$PATH"
   ffmpeg -version
   ffprobe -version
   git clone https://github.com/ggml-org/llama.cpp.git
   cd llama.cpp
   git checkout --detach 620d2225cc436ab0b001e3200e590e3b286ef9ab
   cmake -S . -B build -DCMAKE_BUILD_TYPE=Release -DGGML_METAL=OFF -DGGML_NATIVE=ON -DMTMD_VIDEO=ON -DLLAMA_SUBPROCESS=ON
   cmake --build build --config Release -j 8
   ./build/bin/llama-server --version

   Expected result: Pinned ffmpeg and ffprobe binaries report n8.0.1/source revision; llama-server build succeeds with MTMD_VIDEO and LLAMA_SUBPROCESS enabled. No model inference has run yet.

Download the model

1. 1. Download the pinned Unsloth GGUF and Qwen template

   Verify both public immutable inputs. Qwen base/template and Unsloth quantization have separate revisions and authors. Use the explicit native Jinja template so non-thinking control does not depend on an uninspected embedded-template version. Stop on either hash mismatch.

   CommandCopy command

   cd "$HOME/qwen35-27b-video-hosting"
   curl --fail --location --output models/Qwen3.5-27B-Q4_K_M.gguf 'https://huggingface.co/unsloth/Qwen3.5-27B-GGUF/resolve/3221f178a6b842d04f1fb42f1c413534adcc0a6a/Qwen3.5-27B-Q4_K_M.gguf'
   curl --fail --location --output models/qwen35-chat-template.jinja 'https://huggingface.co/Qwen/Qwen3.5-27B/raw/fc05daec18b0a78c049392ed2e771dde82bdf654/chat_template.jinja'
   sha256sum --check <<'SHA256'
   84b5f7f112156d63836a01a69dc3f11a6ba63b10a23b8ca7a7efaf52d5a2d806  models/Qwen3.5-27B-Q4_K_M.gguf
   a4aee8afcf2e0711942cf848899be66016f8d14a889ff9ede07bca099c28f715  models/qwen35-chat-template.jinja
   SHA256

   Expected result: Both files report OK. Exact Q4_K_M GGUF is 16,740,812,704 bytes and native 27B template 7756 bytes. No weights from another size and no projector are substituted.

2. 2. Download the exact 27B vision projector

   Use the F16 projector from the same exact Unsloth 27B repository revision. Match its SHA-256; do not replace it with a similarly named sibling file.

   CommandCopy command

   cd "$HOME/qwen35-27b-video-hosting"
   curl --fail --location --output models/mmproj-F16.gguf 'https://huggingface.co/unsloth/Qwen3.5-27B-GGUF/resolve/3221f178a6b842d04f1fb42f1c413534adcc0a6a/mmproj-F16.gguf'
   printf '%s  %s\n' '458bc46d8f275866fde5d88c9c554d9d462a6e8e3a028090d9850e17ab6a1217' 'models/mmproj-F16.gguf' | sha256sum --check

   Expected result: mmproj-F16.gguf reports OK; metadata size 927,607,040 bytes. Language-model, native-template and projector checks must all pass.

Configure

1. 1. Create a private service credential

   Generate this only on your own host when following the guide. It is a local service password, not an OpenAI/provider credential. Keep it private and never paste it into tickets. Create it only when absent; preserve an existing key so a repeated setup does not silently rotate access. It is not created during this documentation run.

   CommandCopy command

   cd "$HOME/qwen35-27b-video-hosting"
   umask 077
   if [ ! -e .api-key ]; then
     openssl rand -hex 32 > .api-key
   fi
   test -s .api-key || { echo 'Existing .api-key is empty; restore it before continuing.'; exit 1; }
   chmod 600 .api-key

   Expected result: A non-empty private .api-key file exists; existing key preserved. No secret is printed.

Start the model

1. 1. Start one CPU-only sequence

   Use the same exact 27B language/projector/template pins with explicitly video-enabled build. Server helper uses FFmpeg fps filter at 0.5 fps across the entire input and adds frame timestamps every 1,000 ms; with 2-second sample spacing each frame gets a timestamp. QWEN3VL may merge adjacent frame pairs; do not assume reduction. Conservative five-frame budget at 1,024 tokens each plus 512 output tokens is 5,632; 8,192 leaves 2,560 tokens for prompt/template/timestamps. Actual counts unmeasured. Multimodal runtime disables context shifting; stop on overflow rather than silently change configuration.

   CommandCopy command

   cd "$HOME/qwen35-27b-video-hosting"
   export QWEN35_27B_VIDEO_SERVICE_KEY="$(cat .api-key)"
   ./llama.cpp/build/bin/llama-server -m models/Qwen3.5-27B-Q4_K_M.gguf --alias qwen35-27b --host 127.0.0.1 --port 8080 --api-key "$QWEN35_27B_VIDEO_SERVICE_KEY" --device none -ngl 0 -t 8 -c 8192 -np 1 --cache-type-k f16 --cache-type-v f16 --jinja --chat-template-file models/qwen35-chat-template.jinja --chat-template-kwargs '{"enable_thinking":false}' --reasoning off --reasoning-format none --skip-chat-parsing --temp 0.7 --top-p 0.8 --top-k 20 --min-p 0.0 --presence-penalty 1.5 --repeat-penalty 1.0 --mmproj models/mmproj-F16.gguf --no-mmproj-offload --mmproj-device none --image-min-tokens 1024 --image-max-tokens 1024 --mtmd-batch-max-tokens 1024 --video-fps 0.5 --video-timestamp-interval 1000 --video-ffmpeg-dir "$HOME/qwen35-27b-video-hosting/ffmpeg-bin/bin"

   Expected result: Model initializes and listener starts. Pass health and authenticated chat checks before relying on the service. Command-line credentials may be visible to privileged/local users; trusted host required.

Verify the setup

1. 1. Check readiness and authenticated text generation

   Use a second server terminal and the private service key. Send only plain text. Check final response against the supplied source passage and verify finish_reason is not length; no exact wording is guaranteed. Do not expect reasoning_content in raw mode. If extended thinking appears, non-thinking setup validation failed.

   CommandCopy command

   cd "$HOME/qwen35-27b-video-hosting"
   export QWEN35_27B_VIDEO_SERVICE_KEY="$(cat .api-key)"
   curl --fail http://127.0.0.1:8080/health
   curl --fail http://127.0.0.1:8080/v1/chat/completions -H "Authorization: Bearer $QWEN35_27B_VIDEO_SERVICE_KEY" -H 'Content-Type: application/json' -d '{"model":"qwen35-27b","messages":[{"role":"user","content":"Summarize this supplied passage in one sentence without adding facts: The community library opens at 9am on weekdays and closes at 6pm. On Saturday it opens at 10am and closes at 4pm. It is closed on Sunday."}],"max_tokens":512,"temperature":0.7,"top_k":20,"top_p":0.8,"min_p":0.0,"presence_penalty":1.5,"repeat_penalty":1.0}'
   curl -o /dev/null -s -w '%{http_code}\n' http://127.0.0.1:8080/v1/chat/completions -H 'Content-Type: application/json' -d '{"model":"qwen35-27b","messages":[{"role":"user","content":"Hello"}]}'

   Expected result: Health HTTP 200 when ready; authenticated response contains non-empty choices[0].message.content with an accurate final summary and finish_reason not length, without a generated extended thinking trace. Unauthenticated chat returns HTTP 401. Verify facts yourself; output can be wrong.

2. 2. Verify all sampled frames before video submission

   Download and hash-check the whole immutable moving MP4. ffprobe verifies source metadata. Decode the entire file through the same fixed fps=0.500000 RGB24 filter as the native helper; count all bytes, with no -frames or artificial stop. Reject a changed source, oversized duration, partial frame or any count other than five. This is media decoding only, not model inference.

   CommandCopy command

   cd "$HOME/qwen35-27b-video-hosting"
   export PATH="$HOME/qwen35-27b-video-hosting/ffmpeg-bin/bin:$PATH"
   curl --fail --location --output fixture.mp4 'https://raw.githubusercontent.com/ggml-org/llama.cpp/620d2225cc436ab0b001e3200e590e3b286ef9ab/tools/mtmd/test-3.mp4'
   printf '%s  %s\n' '95a9776a0e7f8d6ab1ba67a31dbb09a4c04ffda3c853fd933624e3496fa970a5' 'fixture.mp4' | sha256sum --check
   ffprobe -v error -select_streams v:0 -show_entries stream=width,height,codec_name:format=duration -of json fixture.mp4 > fixture-metadata.json
   ffmpeg -y -v error -i fixture.mp4 -vf fps=0.500000 -an -f rawvideo -pix_fmt rgb24 fixture-rgb.raw
   python3 - <<'PYPREFLIGHT'
   import json,pathlib
   m=json.loads(pathlib.Path('fixture-metadata.json').read_text())
   s=m['streams'][0]
   assert s['width']==854 and s['height']==358 and s['codec_name']=='h264'
   assert 10.02 <= float(m['format']['duration']) <= 10.04 and float(m['format']['duration']) <= 10.1
   size=pathlib.Path('fixture-rgb.raw').stat().st_size
   frame_bytes=854*358*3
   assert size % frame_bytes == 0, 'partial decoded frame'
   assert size // frame_bytes == 5, 'reject: expected exactly five sampled frames'
   print('Preflight OK: complete five-frame bounded input')
   PYPREFLIGHT

   Expected result: Fixture SHA OK; metadata matches; preflight reports exactly five complete sampled frames. If any assertion fails, stop before video submission. No generated frame limit hides additional decoded frames.

3. 3. Verify grounded temporal video understanding

   Run from the server host or its SSH shell. Activate the same tools by explicit paths. Native request uses input_video.data raw base64, not an image_url still. Inspect the response for temporal order: a man is initially seated under a white cape beside a large gray character; a hand reaches or pulls a chair-side handle; later the man is out of the chair, standing or moving around the room. A static description alone fails. Exact dialogue, names, action timing and prose are not pass criteria; video path does not evaluate audio. Confirm finish_reason is not length and no extended thinking trace. Before the video request, authenticated /props must report modalities.vision=true and modalities.video=true. A successful image load alone is insufficient.

   CommandCopy command

   cd "$HOME/qwen35-27b-video-hosting"
   export QWEN35_27B_VIDEO_SERVICE_KEY="$(cat .api-key)"
   curl --fail http://127.0.0.1:8080/props -H "Authorization: Bearer $QWEN35_27B_VIDEO_SERVICE_KEY" --output video-props.json
   python3 - <<'PYPROPS'
   import json
   p=json.load(open('video-props.json'))
   assert p['modalities']['vision'] and p['modalities']['video'], 'Loaded runtime does not report video capability'
   PYPROPS
   python3 - <<'PYREQUEST'
   import base64,json,pathlib
   video=base64.b64encode(pathlib.Path('fixture.mp4').read_bytes()).decode()
   request={'model':'qwen35-27b','messages':[{'role':'user','content':[{'type':'text','text':'Describe the main physical action in this video in order. Explain how the man and chair change between the beginning, middle and end. Use only visible evidence; do not invent dialogue.'},{'type':'input_video','input_video':{'data':video}}]}],'max_tokens':512,'temperature':0.7,'top_p':0.8,'top_k':20,'min_p':0.0,'presence_penalty':1.5,'repeat_penalty':1.0}
   pathlib.Path('video-request.json').write_text(json.dumps(request))
   PYREQUEST
   curl --fail http://127.0.0.1:8080/v1/chat/completions -H "Authorization: Bearer $QWEN35_27B_VIDEO_SERVICE_KEY" -H 'Content-Type: application/json' --data-binary @video-request.json --output video-response.json
   cat video-response.json

   Expected result: Authenticated HTTP 200 with nonempty choices[0].message.content describing the visible before/after action in correct order, finish_reason not length, no extended thinking. Unsupported-video errors, truncation, incorrect order or invented actions fail. No exact generated response or confidence promised.

Performance evidence

No compatible measurement available. Documented support and estimated memory do not establish response speed.

Limits and troubleshooting

• Documented configuration only. No weights, build, inference, full-fit, throughput or latency benchmark was executed by OpenTools.
• Exact Unsloth Q4_K_M artifact is 16,740,812,704 bytes, derived from Qwen/Qwen3.5-27B at separate revisions. Quantization does not guarantee full-precision quality. No weight-size-only memory fit is claimed.
• The native 27B checkpoint is a dense language backbone with a vision encoder, 64 layers and hidden size 5,120. Its config alternates three Gated DeltaNet layers per full-attention layer; it has 16 linear key heads, 48 linear value heads and four full-attention KV heads. Family MoE claims do not make this checkpoint sparse. Hybrid recurrent state and full-attention cache must not be reduced to a transformer-only memory formula.
• Use Qwen’s non-thinking general-task sampler (temperature 0.7, top-p 0.8, top-k 20, min-p 0, presence penalty 1.5 and repetition penalty 1.0). The model card gives conflicting non-thinking reasoning presets in two sections; this recipe avoids that task/mode and uses the consistent general-task preset.
• Operating system packages and build tool dependencies resolve when installed. Save tool versions/build logs with any actual execution. Cloud hardware is AWS r7i.2xlarge with eight vCPUs and 64 GiB host RAM; eight inference threads are allocated. No exact-instance benchmark is claimed.
• Loopback and a service key do not isolate the service from privileged local users. A shared/public service needs separately designed TLS, authentication and operations.
• The native Transformers config names mamba_ssm_dtype float32, but that does not establish llama.cpp recurrent-state precision or peak memory. This recipe explicitly sets only F16 attention K/V cache; runtime-managed recurrent state remains unmeasured.
• The four-thread, 32 GiB 9B sibling setup is not evidence for this eight-thread, 64 GiB 27B configuration. No throughput, memory or quality observation is transferred from it.
• The native 27B model class Qwen3_5ForConditionalGeneration is explicitly registered by the pinned projector converter, which emits QWEN3VL. Its 27B vision_config has patch size 16, merge size 2 and projection output size 5,120. Pinned mtmd/clip source handles QWEN3VL preprocessing. These sources establish documented mapping, not an executed projector-compatibility result.
• The pinned runtime warns that grounding tasks need at least 1,024 image tokens; this guide uses that nominal budget, but does not validate grounding or fine-print OCR accuracy. Changing resolution/image budget changes the configuration.
• Source fixture duration is about 10.03 seconds. Target 0.5 fps plans five sampled frames; actual decoded frame count is a preflight gate, not a claimed model observation. Native helper has no input duration/frame stop parameter and processes the whole clip. Do not substitute longer videos.
• Temporal frame grouping, image resizing/token bounds, timestamps and template all consume context. 8,192 is an author configuration with conservative reserve, not the full advertised model context. Actual prompt tokens and peak RAM unknown. No long-video/audio benchmark or still-image quality result transfers.
• FFmpeg fps sampling can select source frames near interval midpoints, rather than exact 0, 2, 4, 6 or 8-second seeks. Server timestamps are nominal sampled-frame index divided by target fps, not verified original-source event times. Judge visible action ordering, not exact action timestamps. Video audio is not processed by this configuration.
• Thinking is disabled for this bounded general-task recipe. The 27B publisher recommends at least 128K context to preserve thinking capabilities; this 8K non-thinking setup does not validate those capabilities or long-video understanding.

Download or checksum failure

Verify exact uppercase filename and pinned URL; remove only the incomplete dedicated model file and retry. Do not run a mismatching file.

Architecture unsupported or build fails

Confirm exact pinned source commit, Ubuntu compiler/CMake dependencies and GGUF. Do not substitute an older packaged llama.cpp silently.

Out of memory or context overflow

Stop server, free/provision RAM or reduce context and restart; shorten input/output within configured total. Allocation is not a measured guarantee; record any changed configuration.

Connection refused or HTTP 401

Check foreground server logs, 8080 binding, matching private key and Bearer header. Cloud client must have working SSH tunnel to the correct VM; do not expose the port to repair a tunnel.

Unsupported video or incorrect temporal answer

Confirm matching 27B projector, MTMD_VIDEO=ON and LLAMA_SUBPROCESS=ON, exact FFmpeg tools, input_video.data schema and frame-count preflight. Confirm --video-fps 0.5 and 8,192 context tokens. If logs show unavailable video, overflow or wrong action order stop; do not treat still-image fallback as video support.

Stop the model

1. 1. Stop the service

   Press Ctrl+C in server terminal. Unset key in each shell. Close the SSH tunnel with Ctrl+C. Stopping the process alone does not stop EC2 billing.

   CommandCopy command

   unset QWEN35_27B_VIDEO_SERVICE_KEY

   Expected result: Server stops and localhost:8080 stops serving.

Uninstall

1. 1. Remove dedicated model, build and credential files

   Back up wanted logs/files first. Remove only this walkthrough directory after confirming its path. Shared apt packages can remain; do not remove other workloads.

   CommandCopy command

   rm -r -- "$HOME/qwen35-27b-video-hosting"

   Expected result: Dedicated files and local service credential removed; shared system dependencies remain.

Shut down cloud resources

1. 1. End cloud billing resources

   In EC2 us-east-1, stop for later reuse or terminate this exact VM. Review Delete on termination for its root volume. Delete unwanted detached EBS volumes, snapshots and reserved/Elastic/public IP resources. Stopped VM storage and retained resources may still bill. Check Bills/Cost Explorer later for final usage.

   Expected result: Exact instance is stopped/terminated as intended; no unwanted attached/retained billed resources remain. Instance compute stops only after its lifecycle transition, not when llama-server stops.

Related guides

• https://opentools.ai/llms/qwen35-27b