ERNIE 4.5 21B A3B Thinking

Exact model and requirements

Artifact:

Unsloth (quantizer); Baidu (base model) / ERNIE-4.5-21B-A3B-Thinking Q4_K_M · checkpoint unsloth/ERNIE-4.5-21B-A3B-Thinking-GGUF

Publisher revision:

b1fa62d9e5864da596c78a0fb5cc02e5ffba336c

Artifact digest:

sha256:9cd7c60d3c30f46e7f539043ec19c7ec8953153f65dd9a2a2748b8e3f589d015

Download size:

12.42 GiB

Memory requirement (unknown):

Unknown — No exact full-process peak-memory evidence for this Q4_K_M configuration.

System RAM / disk:

Unknown / Unknown

Placement:

single device. One CPU host; no tensor/pipeline sharding. CPU-only execution; --device none and zero GPU layers. No disk or accelerator offload.

KV cache:

F16 K and V attention cache

Workload:

8,192 token context; Unknown; Unknown; concurrency 1; batch 1; cache unknown; repetitions Unknown.

Tokenizer:

Exact Thinking GGUF tokenizer with independently pinned Baidu native Thinking template · b1fa62d9e5864da596c78a0fb5cc02e5ffba336c; template 4341bb42644d5422859509fa25d41544c57181f8

Sampling / reasoning:

Exact Thinking generation_config: temperature 0.6, top_p 0.95, repetition 1.0 and zero frequency/presence penalties. Author disables extra top_k/min_p filtering with 0; 4,096 output cap includes reasoning and final response. Native template forces think_mode=True and opens <think>. Raw content preserved with skip-chat-parsing/reasoning-format none; final response uses <response> tags. No tool executor or disabled-thinking claim.

Runtime prerequisites:

Ubuntu 24.04 x86_64; See prerequisites above.

Official artifact · License

The exact Baidu Thinking repository includes Apache 2.0 license text; the Unsloth derivative card declares Apache-2.0 and names the Thinking base. Retain license and required notices, identify modifications and review patent/NOTICE conditions for redistribution. This is an Unsloth derivative, not a Baidu-published GGUF.

• ernie-thinking-chat.jinja · 0 GiB · revision 4341bb42644d5422859509fa25d41544c57181f8

• Complete Thinking MoE weights remain resident; 21B total / 3B active is publisher-rounded computation labeling, not a three-billion storage estimate.
• 13,331,018,624-byte Q4_K_M artifact excludes attention cache, buffers, OS and runtime. 32 GiB RAM and 40 GiB disk are planning allocations; full fit is unknown.

Install the runtime

1. 1. Build the pinned CPU server

   Install build dependencies and compile the exact official source revision. These packages are system shared dependencies; no CUDA/GPU driver is required. GGML_NATIVE=ON selects -march=native for this GNU/Linux x86 build, compiling for the host CPU. Build and run on the same machine; no AVX-512 requirement is imposed on the i5-12400.

   CommandCopy command

   sudo apt-get update
   sudo apt-get install -y build-essential cmake git curl ca-certificates openssl
   mkdir -p "$HOME/ernie45-thinking-hosting/models"
   cd "$HOME/ernie45-thinking-hosting"
   git clone https://github.com/ggml-org/llama.cpp.git
   cd llama.cpp
   git checkout --detach 620d2225cc436ab0b001e3200e590e3b286ef9ab
   cmake -S . -B build -DCMAKE_BUILD_TYPE=Release -DGGML_METAL=OFF -DGGML_NATIVE=ON
   cmake --build build --config Release -j 4
   ./build/bin/llama-server --version

   Expected result: Build succeeds and server version identifies pinned source. Stop if checkout or build fails.

Download the model

1. 1. Download exact Thinking weights and native template

   Download both independent immutable inputs and verify exact hashes. Unsloth explicitly names the Thinking base, but does not pin the upstream conversion revision. The selected derivative bytes and native template are exact; runtime startup compatibility remains unexecuted.

   CommandCopy command

   cd "$HOME/ernie45-thinking-hosting"
   curl --fail --location --output models/ERNIE-4.5-21B-A3B-Thinking-Q4_K_M.gguf 'https://huggingface.co/unsloth/ERNIE-4.5-21B-A3B-Thinking-GGUF/resolve/b1fa62d9e5864da596c78a0fb5cc02e5ffba336c/ERNIE-4.5-21B-A3B-Thinking-Q4_K_M.gguf'
   curl --fail --location --output ernie-thinking-chat.jinja 'https://huggingface.co/baidu/ERNIE-4.5-21B-A3B-Thinking/raw/4341bb42644d5422859509fa25d41544c57181f8/chat_template.jinja'
   sha256sum --check <<'SHA256'
   9cd7c60d3c30f46e7f539043ec19c7ec8953153f65dd9a2a2748b8e3f589d015  models/ERNIE-4.5-21B-A3B-Thinking-Q4_K_M.gguf
   e141f14fb656c78182da82e2d900e404c021112a7f2df1953db26cab705c06f0  ernie-thinking-chat.jinja
   SHA256

   Expected result: Both SHA-256 checks report OK. GGUF is 13,331,018,624 bytes; template 3,659 bytes. Stop on mismatch.

Configure

1. 1. Create a private service credential

   Generate this only on your own host when following the guide. It is a local service password, not an OpenAI/provider credential. Keep it private and never paste it into tickets. Create it only when absent; preserve an existing key so a repeated setup does not silently rotate access. It is not created during this documentation run.

   CommandCopy command

   cd "$HOME/ernie45-thinking-hosting"
   umask 077
   if [ ! -e .api-key ]; then
     openssl rand -hex 32 > .api-key
   fi
   test -s .api-key || { echo 'Existing .api-key is empty; restore it before continuing.'; exit 1; }
   chmod 600 .api-key

   Expected result: A non-empty private .api-key file exists; existing key preserved. No secret is printed.

Start the model

1. 1. Start one bounded CPU Thinking sequence

   The exact Thinking config matches the pinned native ERNIE4.5 MoE architecture. Use its native template verbatim: system think_mode=True, assistant <think> prefill and final <response> wrapper. Preserve raw tags; do not use the non-Thinking template or a generic reasoning parser. Total context is 8,192 tokens with up to 4,096 generated reasoning-plus-final tokens; completion is not guaranteed.

   CommandCopy command

   cd "$HOME/ernie45-thinking-hosting"
   export ERNIE45_THINK_SERVICE_KEY="$(cat .api-key)"
   ./llama.cpp/build/bin/llama-server -m models/ERNIE-4.5-21B-A3B-Thinking-Q4_K_M.gguf --alias ernie45-thinking --host 127.0.0.1 --port 8080 --api-key "$ERNIE45_THINK_SERVICE_KEY" --device none -ngl 0 -t 4 -c 8192 -np 1 --cache-type-k f16 --cache-type-v f16 --jinja --chat-template-file ernie-thinking-chat.jinja --skip-chat-parsing --reasoning-format none --temp 0.6 --top-k 0 --top-p 0.95 --min-p 0.0 --repeat-penalty 1.0 --frequency-penalty 0.0 --presence-penalty 0.0

   Expected result: Model initializes and listener starts. Pass health and authenticated chat checks before relying on the service. Command-line credentials may be visible to privileged/local users; trusted host required.

Verify the setup

1. 1. Check readiness, complete thinking and a grounded final response

   In another server terminal, check authenticated generation. The opening <think> is prefilled and may not repeat in returned content. Require a closing </think> and a nonempty closed <response>...</response> final body whose total is 36, with finish_reason other than length. Missing closure, wrong total, truncation or reasoning-only content fails the smoke check. This is a pass criterion, not an observed result.

   CommandCopy command

   cd "$HOME/ernie45-thinking-hosting"
   export ERNIE45_THINK_SERVICE_KEY="$(cat .api-key)"
   curl --fail http://127.0.0.1:8080/health
   curl --fail http://127.0.0.1:8080/v1/chat/completions -H "Authorization: Bearer $ERNIE45_THINK_SERVICE_KEY" -H 'Content-Type: application/json' --data-binary '{"model":"ernie45-thinking","messages":[{"role":"user","content":"A box contains 12 red counters, 18 blue counters and 6 green counters. How many counters are in the box? Think carefully and give a short final answer."}],"max_tokens":4096,"temperature":0.6,"top_k":0,"top_p":0.95,"min_p":0.0,"repeat_penalty":1.0,"frequency_penalty":0.0,"presence_penalty":0.0}' --output thinking-response.json
   cat thinking-response.json
   curl -s -o /dev/null -w '%{http_code}\n' http://127.0.0.1:8080/v1/chat/completions -H 'Content-Type: application/json' --data-binary '{"model":"ernie45-thinking","messages":[{"role":"user","content":"Hello"}]}'

   Expected result: Health HTTP 200; authorized raw message.content includes completed thinking and a closed final response with total 36, finish_reason not length; missing key HTTP 401. No inference result was observed here.

Performance evidence

No compatible measurement available. Documented support and estimated memory do not establish response speed.

Limits and troubleshooting

• Documented configuration only: no weights, build, inference or process-memory/speed/quality measurements performed.
• The exact Thinking checkpoint is text MoE with publisher-rounded 21B total / 3B active. All full experts matter for CPU residency. Do not substitute non-Thinking PT or Base weights/template.
• Own config supports 131,072 positions. This deliberate 8,192 combined context / 4,096 output smoke workload is smaller, may truncate increased-length thinking and does not establish adequate budget for complex reasoning.
• The native template hardcodes think_mode=True and ends with <think> prefill. Responses conventionally use closed </think> then <response>...</response>. Raw output is returned in message.content; no separate reasoning_content or tool execution is assumed.
• Publisher generation_config supplies .6 temperature/.95 top_p/repetition1/frequency0/presence0. Extra top-k/min-p filtering is disabled explicitly by author choice.
• A closed correct final response and nonlength finish are required reader checks, not a guarantee. Reasoning-only or malformed/truncated responses fail; changing budget/context creates a different configuration.
• The Unsloth derivative metadata links the Thinking base but not a source conversion revision. Runnable GGUF and native template are separately pinned; no exact-source-conversion or quantization-quality proof is claimed.
• 32 GiB RAM, 40 GiB disk and four threads are allocations, not measured minima or fit. F16 attention K/V cache does not determine full process peak.
• Publisher source also describes tool usage, FastDeploy 2.2 with one 80 GB GPU and parsers, and long-context work. Those useful jobs require independent runtime/hardware/parser qualification; this CPU text pair does not resolve or exclude them.
• Loopback and private service key assume a trusted host. Public/shared serving needs separately designed TLS, isolation and operations; dependencies resolve at installation and versions/logs should be preserved.

Checksum or template failure

Stop on mismatched Thinking GGUF/template hashes. Use exact pinned native Thinking template; do not substitute the earlier PT template.

Unclosed thinking, absent final response or length finish

Treat the check as failed. Require closed thinking and a nonempty closed response wrapper with accurate facts. Do not strip partial traces into a final answer. Any larger output/context allocation is a changed configuration requiring validation.

Out of memory or context overflow

Stop and inspect resource usage. Allocation is not a measured guarantee; shorten workload or explicitly select more resources. All prompt/template/thinking/final output share the context.

HTTP 401 or connection refused

Check foreground startup, loopback 8080 and matching private key. Run the client on this same desktop.

Unexpected tool markup or poor result

This guide configures no tools or tool parser. Use a plain user prompt, inspect raw response, and validate the final content. Publisher capability claims do not establish exact CPU job reliability.

Stop the model

1. 1. Stop the service

   Press Ctrl+C in server terminal. Unset key in each shell. Confirm this desktop localhost:8080 no longer serves the model.

   CommandCopy command

   unset ERNIE45_THINK_SERVICE_KEY

   Expected result: Server stops and localhost:8080 stops serving.

Uninstall

1. 1. Remove dedicated model, build and credential files

   Back up wanted logs/files first. Remove only this walkthrough directory after confirming its path. Shared apt packages can remain; do not remove other workloads.

   CommandCopy command

   rm -r -- "$HOME/ernie45-thinking-hosting"

   Expected result: Dedicated files and local service credential removed; shared system dependencies remain.

Related guides

• https://opentools.ai/llms/ernie-45-21b-a3b-thinking

Exact model and requirements

Artifact:

Unsloth (quantizer); Baidu (base model) / ERNIE-4.5-21B-A3B-Thinking Q4_K_M · checkpoint unsloth/ERNIE-4.5-21B-A3B-Thinking-GGUF

Publisher revision:

b1fa62d9e5864da596c78a0fb5cc02e5ffba336c

Artifact digest:

sha256:9cd7c60d3c30f46e7f539043ec19c7ec8953153f65dd9a2a2748b8e3f589d015

Download size:

12.42 GiB

Memory requirement (unknown):

Unknown — No exact full-process peak-memory evidence for this Q4_K_M configuration.

System RAM / disk:

Unknown / Unknown

Placement:

single device. One CPU host; no tensor/pipeline sharding. CPU-only execution; --device none and zero GPU layers. No disk or accelerator offload.

KV cache:

F16 K and V attention cache

Workload:

8,192 token context; Unknown; Unknown; concurrency 1; batch 1; cache unknown; repetitions Unknown.

Tokenizer:

Exact Thinking GGUF tokenizer with independently pinned Baidu native Thinking template · b1fa62d9e5864da596c78a0fb5cc02e5ffba336c; template 4341bb42644d5422859509fa25d41544c57181f8

Sampling / reasoning:

Exact Thinking generation_config: temperature 0.6, top_p 0.95, repetition 1.0 and zero frequency/presence penalties. Author disables extra top_k/min_p filtering with 0; 4,096 output cap includes reasoning and final response. Native template forces think_mode=True and opens <think>. Raw content preserved with skip-chat-parsing/reasoning-format none; final response uses <response> tags. No tool executor or disabled-thinking claim.

Runtime prerequisites:

Ubuntu 24.04 x86_64; See prerequisites above.

Official artifact · License

The exact Baidu Thinking repository includes Apache 2.0 license text; the Unsloth derivative card declares Apache-2.0 and names the Thinking base. Retain license and required notices, identify modifications and review patent/NOTICE conditions for redistribution. This is an Unsloth derivative, not a Baidu-published GGUF.

• ernie-thinking-chat.jinja · 0 GiB · revision 4341bb42644d5422859509fa25d41544c57181f8

• Complete Thinking MoE weights remain resident; 21B total / 3B active is publisher-rounded computation labeling, not a three-billion storage estimate.
• 13,331,018,624-byte Q4_K_M artifact excludes attention cache, buffers, OS and runtime. 32 GiB RAM and 40 GiB disk are planning allocations; full fit is unknown.

Install the runtime

1. 1. Build the pinned CPU server

   Install build dependencies and compile the exact official source revision. These packages are system shared dependencies; no CUDA/GPU driver is required. GGML_NATIVE=ON selects -march=native for this GNU/Linux x86 build, compiling for the host CPU. Build and run on the same machine; the binary uses the instruction set exposed by this r7i.xlarge guest. Do not copy it to a different instance family without rebuilding.

   CommandCopy command

   sudo apt-get update
   sudo apt-get install -y build-essential cmake git curl ca-certificates openssl
   mkdir -p "$HOME/ernie45-thinking-hosting/models"
   cd "$HOME/ernie45-thinking-hosting"
   git clone https://github.com/ggml-org/llama.cpp.git
   cd llama.cpp
   git checkout --detach 620d2225cc436ab0b001e3200e590e3b286ef9ab
   cmake -S . -B build -DCMAKE_BUILD_TYPE=Release -DGGML_METAL=OFF -DGGML_NATIVE=ON
   cmake --build build --config Release -j 4
   ./build/bin/llama-server --version

   Expected result: Build succeeds and server version identifies pinned source. Stop if checkout or build fails.

Download the model

1. 1. Download exact Thinking weights and native template

   Download both independent immutable inputs and verify exact hashes. Unsloth explicitly names the Thinking base, but does not pin the upstream conversion revision. The selected derivative bytes and native template are exact; runtime startup compatibility remains unexecuted.

   CommandCopy command

   cd "$HOME/ernie45-thinking-hosting"
   curl --fail --location --output models/ERNIE-4.5-21B-A3B-Thinking-Q4_K_M.gguf 'https://huggingface.co/unsloth/ERNIE-4.5-21B-A3B-Thinking-GGUF/resolve/b1fa62d9e5864da596c78a0fb5cc02e5ffba336c/ERNIE-4.5-21B-A3B-Thinking-Q4_K_M.gguf'
   curl --fail --location --output ernie-thinking-chat.jinja 'https://huggingface.co/baidu/ERNIE-4.5-21B-A3B-Thinking/raw/4341bb42644d5422859509fa25d41544c57181f8/chat_template.jinja'
   sha256sum --check <<'SHA256'
   9cd7c60d3c30f46e7f539043ec19c7ec8953153f65dd9a2a2748b8e3f589d015  models/ERNIE-4.5-21B-A3B-Thinking-Q4_K_M.gguf
   e141f14fb656c78182da82e2d900e404c021112a7f2df1953db26cab705c06f0  ernie-thinking-chat.jinja
   SHA256

   Expected result: Both SHA-256 checks report OK. GGUF is 13,331,018,624 bytes; template 3,659 bytes. Stop on mismatch.

Configure

1. 1. Create a private service credential

   Generate this only on your own host when following the guide. It is a local service password, not an OpenAI/provider credential. Keep it private and never paste it into tickets. Create it only when absent; preserve an existing key so a repeated setup does not silently rotate access. It is not created during this documentation run.

   CommandCopy command

   cd "$HOME/ernie45-thinking-hosting"
   umask 077
   if [ ! -e .api-key ]; then
     openssl rand -hex 32 > .api-key
   fi
   test -s .api-key || { echo 'Existing .api-key is empty; restore it before continuing.'; exit 1; }
   chmod 600 .api-key

   Expected result: A non-empty private .api-key file exists; existing key preserved. No secret is printed.

Start the model

1. 1. Start one bounded CPU Thinking sequence

   The exact Thinking config matches the pinned native ERNIE4.5 MoE architecture. Use its native template verbatim: system think_mode=True, assistant <think> prefill and final <response> wrapper. Preserve raw tags; do not use the non-Thinking template or a generic reasoning parser. Total context is 8,192 tokens with up to 4,096 generated reasoning-plus-final tokens; completion is not guaranteed.

   CommandCopy command

   cd "$HOME/ernie45-thinking-hosting"
   export ERNIE45_THINK_SERVICE_KEY="$(cat .api-key)"
   ./llama.cpp/build/bin/llama-server -m models/ERNIE-4.5-21B-A3B-Thinking-Q4_K_M.gguf --alias ernie45-thinking --host 127.0.0.1 --port 8080 --api-key "$ERNIE45_THINK_SERVICE_KEY" --device none -ngl 0 -t 4 -c 8192 -np 1 --cache-type-k f16 --cache-type-v f16 --jinja --chat-template-file ernie-thinking-chat.jinja --skip-chat-parsing --reasoning-format none --temp 0.6 --top-k 0 --top-p 0.95 --min-p 0.0 --repeat-penalty 1.0 --frequency-penalty 0.0 --presence-penalty 0.0

   Expected result: Model initializes and listener starts. Pass health and authenticated chat checks before relying on the service. Command-line credentials may be visible to privileged/local users; trusted host required.

Verify the setup

1. 1. Check readiness, complete thinking and a grounded final response

   In another server terminal, check authenticated generation. The opening <think> is prefilled and may not repeat in returned content. Require a closing </think> and a nonempty closed <response>...</response> final body whose total is 36, with finish_reason other than length. Missing closure, wrong total, truncation or reasoning-only content fails the smoke check. This is a pass criterion, not an observed result.

   CommandCopy command

   cd "$HOME/ernie45-thinking-hosting"
   export ERNIE45_THINK_SERVICE_KEY="$(cat .api-key)"
   curl --fail http://127.0.0.1:8080/health
   curl --fail http://127.0.0.1:8080/v1/chat/completions -H "Authorization: Bearer $ERNIE45_THINK_SERVICE_KEY" -H 'Content-Type: application/json' --data-binary '{"model":"ernie45-thinking","messages":[{"role":"user","content":"A box contains 12 red counters, 18 blue counters and 6 green counters. How many counters are in the box? Think carefully and give a short final answer."}],"max_tokens":4096,"temperature":0.6,"top_k":0,"top_p":0.95,"min_p":0.0,"repeat_penalty":1.0,"frequency_penalty":0.0,"presence_penalty":0.0}' --output thinking-response.json
   cat thinking-response.json
   curl -s -o /dev/null -w '%{http_code}\n' http://127.0.0.1:8080/v1/chat/completions -H 'Content-Type: application/json' --data-binary '{"model":"ernie45-thinking","messages":[{"role":"user","content":"Hello"}]}'

   Expected result: Health HTTP 200; authorized raw message.content includes completed thinking and a closed final response with total 36, finish_reason not length; missing key HTTP 401. No inference result was observed here.

Performance evidence

No compatible measurement available. Documented support and estimated memory do not establish response speed.

Limits and troubleshooting

• Documented configuration only: no weights, build, inference or process-memory/speed/quality measurements performed.
• The exact Thinking checkpoint is text MoE with publisher-rounded 21B total / 3B active. All full experts matter for CPU residency. Do not substitute non-Thinking PT or Base weights/template.
• Own config supports 131,072 positions. This deliberate 8,192 combined context / 4,096 output smoke workload is smaller, may truncate increased-length thinking and does not establish adequate budget for complex reasoning.
• The native template hardcodes think_mode=True and ends with <think> prefill. Responses conventionally use closed </think> then <response>...</response>. Raw output is returned in message.content; no separate reasoning_content or tool execution is assumed.
• Publisher generation_config supplies .6 temperature/.95 top_p/repetition1/frequency0/presence0. Extra top-k/min-p filtering is disabled explicitly by author choice.
• A closed correct final response and nonlength finish are required reader checks, not a guarantee. Reasoning-only or malformed/truncated responses fail; changing budget/context creates a different configuration.
• The Unsloth derivative metadata links the Thinking base but not a source conversion revision. Runnable GGUF and native template are separately pinned; no exact-source-conversion or quantization-quality proof is claimed.
• 32 GiB RAM, 40 GiB disk and four threads are allocations, not measured minima or fit. F16 attention K/V cache does not determine full process peak.
• Publisher source also describes tool usage, FastDeploy 2.2 with one 80 GB GPU and parsers, and long-context work. Those useful jobs require independent runtime/hardware/parser qualification; this CPU text pair does not resolve or exclude them.
• Loopback and private service key assume a trusted host. Public/shared serving needs separately designed TLS, isolation and operations; dependencies resolve at installation and versions/logs should be preserved.

Checksum or template failure

Stop on mismatched Thinking GGUF/template hashes. Use exact pinned native Thinking template; do not substitute the earlier PT template.

Unclosed thinking, absent final response or length finish

Treat the check as failed. Require closed thinking and a nonempty closed response wrapper with accurate facts. Do not strip partial traces into a final answer. Any larger output/context allocation is a changed configuration requiring validation.

Out of memory or context overflow

Stop and inspect resource usage. Allocation is not a measured guarantee; shorten workload or explicitly select more resources. All prompt/template/thinking/final output share the context.

HTTP 401 or connection refused

Check foreground startup, loopback 8080 and matching private key. Verify the SSH tunnel targets the intended VM; keep public API ingress closed.

Unexpected tool markup or poor result

This guide configures no tools or tool parser. Use a plain user prompt, inspect raw response, and validate the final content. Publisher capability claims do not establish exact CPU job reliability.

Stop the model

1. 1. Stop the service

   Press Ctrl+C in server terminal. Unset key in each shell. Close the SSH tunnel with Ctrl+C. Stopping the process alone does not stop EC2 billing.

   CommandCopy command

   unset ERNIE45_THINK_SERVICE_KEY

   Expected result: Server stops and localhost:8080 stops serving.

Uninstall

1. 1. Remove dedicated model, build and credential files

   Back up wanted logs/files first. Remove only this walkthrough directory after confirming its path. Shared apt packages can remain; do not remove other workloads.

   CommandCopy command

   rm -r -- "$HOME/ernie45-thinking-hosting"

   Expected result: Dedicated files and local service credential removed; shared system dependencies remain.

Shut down cloud resources

1. 1. End cloud billing resources

   In EC2 us-east-1, stop for later reuse or terminate this exact VM. Review Delete on termination for its root volume. Delete unwanted detached EBS volumes, snapshots and reserved/Elastic/public IP resources. Stopped VM storage and retained resources may still bill. Check Bills/Cost Explorer later for final usage.

   Expected result: Exact instance is stopped/terminated as intended; no unwanted attached/retained billed resources remain. Instance compute stops only after its lifecycle transition, not when llama-server stops.

Related guides

• https://opentools.ai/llms/ernie-45-21b-a3b-thinking