ERNIE 4.5 21B A3B

Exact model and requirements

Artifact:

Unsloth (quantizer); Baidu (base model) / ERNIE-4.5-21B-A3B-PT Q4_K_M · checkpoint unsloth/ERNIE-4.5-21B-A3B-PT-GGUF

Publisher revision:

de45a831ceac3c876bb63f39bc60406a7d07e3bb

Artifact digest:

sha256:eac44e96282fee37140f82f435d4c37c4ec0edd42fe5d93f09e6ed8786a6cff0

Download size:

12.42 GiB

Memory requirement (unknown):

Unknown — No exact full-process peak-memory evidence for this Q4_K_M configuration.

System RAM / disk:

Unknown / Unknown

Placement:

single device. One CPU host; no tensor/pipeline sharding. CPU-only execution; --device none and zero GPU layers. No disk or accelerator offload.

KV cache:

F16 K and V attention cache

Workload:

4,096 token context; Unknown; Unknown; concurrency 1; batch 1; cache unknown; repetitions Unknown.

Tokenizer:

Embedded ERNIE GGUF tokenizer with exact pinned Baidu PT chat template override · 87db95487941cb39592ee0abca3b9155a6d19c5c

Sampling / reasoning:

temperature 0.8, top_p 0.8, repeat_penalty 1.0, frequency/presence penalties 0; top_k 0 and min_p 0 disable extra filters. Validation output cap 128; author short workload. Non-Thinking post-trained text checkpoint. Native User:/Assistant: template, no thinking parser, tools or multimodal input.

Runtime prerequisites:

Ubuntu 24.04 x86_64; See prerequisites above.

Official artifact · License

Baidu publisher and Unsloth derivative metadata declare Apache 2.0. Preserve the license and required notices, identify modifications and review patent/NOTICE conditions for redistribution. This is an Unsloth quantization, not a Baidu-published GGUF.

• The complete 21B-total-parameter MoE artifact includes routed and shared expert weights; 3B active per token does not imply 3B storage or RAM.
• The 13,331,015,456-byte file excludes runtime buffers, attention cache, OS and other allocations. 32 GiB RAM and 40 GiB disk are author resource choices, not measured minimums or fit guarantees.

Install the runtime

1. 1. Build the pinned CPU server

   Install build dependencies and compile the exact official source revision. These packages are system shared dependencies; no CUDA/GPU driver is required. GGML_NATIVE=ON selects -march=native for this GNU/Linux x86 build, compiling for the host CPU. Build and run on the same machine; no AVX-512 requirement is imposed on the i5-12400.

   CommandCopy command

   sudo apt-get update
   sudo apt-get install -y build-essential cmake git curl ca-certificates openssl
   mkdir -p "$HOME/ernie45-hosting/models"
   cd "$HOME/ernie45-hosting"
   git clone https://github.com/ggml-org/llama.cpp.git
   cd llama.cpp
   git checkout --detach 620d2225cc436ab0b001e3200e590e3b286ef9ab
   cmake -S . -B build -DCMAKE_BUILD_TYPE=Release -DGGML_METAL=OFF -DGGML_NATIVE=ON
   cmake --build build --config Release -j 4
   ./build/bin/llama-server --version

   Expected result: Build succeeds and server version identifies pinned source. Stop if checkout or build fails.

Download the model

1. 1. Download and verify the exact GGUF

   Download the complete Unsloth derivative, then fetch the independently pinned Baidu chat template. Metadata reports public ungated repositories. Do not substitute Thinking, Base or Paddle weights.

   CommandCopy command

   cd "$HOME/ernie45-hosting"
   curl --fail --location --output models/ERNIE-4.5-21B-A3B-PT-Q4_K_M.gguf 'https://huggingface.co/unsloth/ERNIE-4.5-21B-A3B-PT-GGUF/resolve/de45a831ceac3c876bb63f39bc60406a7d07e3bb/ERNIE-4.5-21B-A3B-PT-Q4_K_M.gguf'
   printf '%s  %s\n' 'eac44e96282fee37140f82f435d4c37c4ec0edd42fe5d93f09e6ed8786a6cff0' 'models/ERNIE-4.5-21B-A3B-PT-Q4_K_M.gguf' | sha256sum --check
   curl --fail --location --output ernie-chat.jinja 'https://huggingface.co/baidu/ERNIE-4.5-21B-A3B-PT/raw/87db95487941cb39592ee0abca3b9155a6d19c5c/chat_template.jinja'
   printf '%s  %s\n' '471fb571976770318fe6f27749e37940be8d29c0a7770558140f141f5906f2a2' 'ernie-chat.jinja' | sha256sum --check

   Expected result: Both SHA-256 checks report OK; GGUF size 13,331,015,456 bytes. Stop on mismatch.

Configure

1. 1. Create a private service credential

   Generate this only on your own host when following the guide. It is a local service password, not an OpenAI/provider credential. Keep it private and never paste it into tickets. Create it only when absent; preserve an existing key so a repeated setup does not silently rotate access. It is not created during this documentation run.

   CommandCopy command

   cd "$HOME/ernie45-hosting"
   umask 077
   if [ ! -e .api-key ]; then
     openssl rand -hex 32 > .api-key
   fi
   test -s .api-key || { echo 'Existing .api-key is empty; restore it before continuing.'; exit 1; }
   chmod 600 .api-key

   Expected result: A non-empty private .api-key file exists; existing key preserved. No secret is printed.

Start the model

1. 1. Start one CPU-only sequence

   Use the native ERNIE4.5 MoE implementation at the pinned runtime revision. Force the exact Baidu User:/Assistant: chat template through Jinja, with one CPU slot and 4,096 total context tokens. Reserve 128 tokens for output. No thinking parser, tool executor or accelerator is configured.

   CommandCopy command

   cd "$HOME/ernie45-hosting"
   export ERNIE45_SERVICE_KEY="$(cat .api-key)"
   ./llama.cpp/build/bin/llama-server -m models/ERNIE-4.5-21B-A3B-PT-Q4_K_M.gguf --alias ernie45-21ba3b --host 127.0.0.1 --port 8080 --api-key "$ERNIE45_SERVICE_KEY" --device none -ngl 0 -t 4 -c 4096 -np 1 --cache-type-k f16 --cache-type-v f16 --jinja --chat-template-file ernie-chat.jinja --reasoning off --temp 0.8 --top-k 0 --top-p 0.8 --min-p 0.0 --repeat-penalty 1.0 --frequency-penalty 0.0 --presence-penalty 0.0

   Expected result: Model initializes and listener starts. Pass health and authenticated chat checks before relying on the service. Command-line credentials may be visible to privileged/local users; trusted host required.

Verify the setup

1. 1. Check readiness and authenticated text generation

   Run checks from a second terminal on the server. Health is public on loopback. Check the authenticated chat’s non-empty final content against the supplied passage and ensure finish_reason is not length; no exact wording is guaranteed. This is a short functional summarization check, not a quality benchmark.

   CommandCopy command

   cd "$HOME/ernie45-hosting"
   export ERNIE45_SERVICE_KEY="$(cat .api-key)"
   curl --fail http://127.0.0.1:8080/health
   curl --fail http://127.0.0.1:8080/v1/chat/completions -H "Authorization: Bearer $ERNIE45_SERVICE_KEY" -H 'Content-Type: application/json' -d '{"model":"ernie45-21ba3b","messages":[{"role":"user","content":"Summarize this supplied passage in one sentence without adding facts: The community library opens at 9am on weekdays and closes at 6pm. On Saturday it opens at 10am and closes at 4pm. It is closed on Sunday."}],"max_tokens":128,"temperature":0.8,"top_k":0,"top_p":0.8,"min_p":0.0,"repeat_penalty":1.0}'
   curl -o /dev/null -s -w '%{http_code}\n' http://127.0.0.1:8080/v1/chat/completions -H 'Content-Type: application/json' -d '{"model":"ernie45-21ba3b","messages":[{"role":"user","content":"Hello"}]}'

   Expected result: Health HTTP 200 when ready; authorized response contains non-empty choices[0].message.content accurately summarizing the supplied opening hours, with finish_reason not length; unauthorized chat returns HTTP 401. Inspect facts yourself; generated text can be wrong.

Performance evidence

No compatible measurement available. Documented support and estimated memory do not establish response speed.

Limits and troubleshooting

• Documented architecture support only. Exact full-process memory, fit, latency, throughput and answer quality are unknown. No model execution occurred.
• This is the text post-trained non-Thinking checkpoint. Publisher overview lists 21B total / 3B active per token. Training-family multimodal discussion does not make this text model a vision guide.
• The source config supports 131,072 positions; this recipe deliberately configures 4,096 combined prompt/output tokens, one sequence and a 128-token validation output cap.
• Baidu generation_config gives temperature 0.8, top_p 0.8, repetition penalty 1.0 and zero frequency/presence penalties. Additional top-k/min-p filtering is disabled explicitly. No sibling sampler or speed is transferred.
• Native template emits begin-of-sentence, User:, Assistant: and end-of-sentence tokens. The pinned original template is explicitly loaded; do not replace it with a generic or Thinking template.
• 32 GiB RAM and 40 GiB disk are planning allocations, not measured requirements. Full experts and runtime state remain relevant despite 3B active computation.
• System packages and build dependencies resolve at installation. Save versions and build logs on execution; build GGML_NATIVE on the same CPU used for inference.
• The quantizer is Unsloth; its own derivative card and metadata link the Baidu base and Apache 2.0 license. Quantization quality has not been measured here.
• Loopback and a private service key require a trusted host. Shared/public production access needs separately designed isolation, TLS and operations.
• Whole-model coverage is not final; this pair documents one useful private CPU text configuration.

Download or checksum failure

Check exact pinned Unsloth GGUF and Baidu template URLs, filename case and both hashes. Do not serve mismatched bytes.

Unsupported architecture or build failure

Use exact llama.cpp commit 620d2225cc436ab0b001e3200e590e3b286ef9ab with native ERNIE4.5 MoE support. Do not silently substitute an older package.

Out of memory or context overflow

Stop the service and inspect host memory. Shorten prompt/output within 4,096 total tokens or explicitly change allocation. Full fit remains unknown.

HTTP 401 or connection refused

Check foreground server startup, loopback port 8080 and matching Bearer service key. Run the client on the same desktop.

Poor or truncated summary

Confirm non-Thinking PT-derived GGUF, exact publisher template and sampler. Inspect finish_reason and supplied facts. Increase output cap only within total context and record the changed workload; smoke checks do not establish reliable quality.

Stop the model

1. 1. Stop the service

   Press Ctrl+C in server terminal. Unset key in each shell. Confirm this desktop localhost:8080 no longer serves the model.

   CommandCopy command

   unset ERNIE45_SERVICE_KEY

   Expected result: Server stops and localhost:8080 stops serving.

Uninstall

1. 1. Remove dedicated model, build and credential files

   Back up wanted logs/files first. Remove only this walkthrough directory after confirming its path. Shared apt packages can remain; do not remove other workloads.

   CommandCopy command

   rm -r -- "$HOME/ernie45-hosting"

   Expected result: Dedicated files and local service credential removed; shared system dependencies remain.

Related guides

• https://opentools.ai/llms/ernie-45-21b-a3b

Exact model and requirements

Artifact:

Unsloth (quantizer); Baidu (base model) / ERNIE-4.5-21B-A3B-PT Q4_K_M · checkpoint unsloth/ERNIE-4.5-21B-A3B-PT-GGUF

Publisher revision:

de45a831ceac3c876bb63f39bc60406a7d07e3bb

Artifact digest:

sha256:eac44e96282fee37140f82f435d4c37c4ec0edd42fe5d93f09e6ed8786a6cff0

Download size:

12.42 GiB

Memory requirement (unknown):

Unknown — No exact full-process peak-memory evidence for this Q4_K_M configuration.

System RAM / disk:

Unknown / Unknown

Placement:

single device. One CPU host; no tensor/pipeline sharding. CPU-only execution; --device none and zero GPU layers. No disk or accelerator offload.

KV cache:

F16 K and V attention cache

Workload:

4,096 token context; Unknown; Unknown; concurrency 1; batch 1; cache unknown; repetitions Unknown.

Tokenizer:

Embedded ERNIE GGUF tokenizer with exact pinned Baidu PT chat template override · 87db95487941cb39592ee0abca3b9155a6d19c5c

Sampling / reasoning:

temperature 0.8, top_p 0.8, repeat_penalty 1.0, frequency/presence penalties 0; top_k 0 and min_p 0 disable extra filters. Validation output cap 128; author short workload. Non-Thinking post-trained text checkpoint. Native User:/Assistant: template, no thinking parser, tools or multimodal input.

Runtime prerequisites:

Ubuntu 24.04 x86_64; See prerequisites above.

Official artifact · License

Baidu publisher and Unsloth derivative metadata declare Apache 2.0. Preserve the license and required notices, identify modifications and review patent/NOTICE conditions for redistribution. This is an Unsloth quantization, not a Baidu-published GGUF.

• The complete 21B-total-parameter MoE artifact includes routed and shared expert weights; 3B active per token does not imply 3B storage or RAM.
• The 13,331,015,456-byte file excludes runtime buffers, attention cache, OS and other allocations. 32 GiB RAM and 40 GiB disk are author resource choices, not measured minimums or fit guarantees.

Install the runtime

1. 1. Build the pinned CPU server

   Install build dependencies and compile the exact official source revision. These packages are system shared dependencies; no CUDA/GPU driver is required. GGML_NATIVE=ON selects -march=native for this GNU/Linux x86 build, compiling for the host CPU. Build and run on the same machine; the binary uses the instruction set exposed by this r7i.xlarge guest. Do not copy it to a different instance family without rebuilding.

   CommandCopy command

   sudo apt-get update
   sudo apt-get install -y build-essential cmake git curl ca-certificates openssl
   mkdir -p "$HOME/ernie45-hosting/models"
   cd "$HOME/ernie45-hosting"
   git clone https://github.com/ggml-org/llama.cpp.git
   cd llama.cpp
   git checkout --detach 620d2225cc436ab0b001e3200e590e3b286ef9ab
   cmake -S . -B build -DCMAKE_BUILD_TYPE=Release -DGGML_METAL=OFF -DGGML_NATIVE=ON
   cmake --build build --config Release -j 4
   ./build/bin/llama-server --version

   Expected result: Build succeeds and server version identifies pinned source. Stop if checkout or build fails.

Download the model

1. 1. Download and verify the exact GGUF

   Download the complete Unsloth derivative, then fetch the independently pinned Baidu chat template. Metadata reports public ungated repositories. Do not substitute Thinking, Base or Paddle weights.

   CommandCopy command

   cd "$HOME/ernie45-hosting"
   curl --fail --location --output models/ERNIE-4.5-21B-A3B-PT-Q4_K_M.gguf 'https://huggingface.co/unsloth/ERNIE-4.5-21B-A3B-PT-GGUF/resolve/de45a831ceac3c876bb63f39bc60406a7d07e3bb/ERNIE-4.5-21B-A3B-PT-Q4_K_M.gguf'
   printf '%s  %s\n' 'eac44e96282fee37140f82f435d4c37c4ec0edd42fe5d93f09e6ed8786a6cff0' 'models/ERNIE-4.5-21B-A3B-PT-Q4_K_M.gguf' | sha256sum --check
   curl --fail --location --output ernie-chat.jinja 'https://huggingface.co/baidu/ERNIE-4.5-21B-A3B-PT/raw/87db95487941cb39592ee0abca3b9155a6d19c5c/chat_template.jinja'
   printf '%s  %s\n' '471fb571976770318fe6f27749e37940be8d29c0a7770558140f141f5906f2a2' 'ernie-chat.jinja' | sha256sum --check

   Expected result: Both SHA-256 checks report OK; GGUF size 13,331,015,456 bytes. Stop on mismatch.

Configure

1. 1. Create a private service credential

   Generate this only on your own host when following the guide. It is a local service password, not an OpenAI/provider credential. Keep it private and never paste it into tickets. Create it only when absent; preserve an existing key so a repeated setup does not silently rotate access. It is not created during this documentation run.

   CommandCopy command

   cd "$HOME/ernie45-hosting"
   umask 077
   if [ ! -e .api-key ]; then
     openssl rand -hex 32 > .api-key
   fi
   test -s .api-key || { echo 'Existing .api-key is empty; restore it before continuing.'; exit 1; }
   chmod 600 .api-key

   Expected result: A non-empty private .api-key file exists; existing key preserved. No secret is printed.

Start the model

1. 1. Start one CPU-only sequence

   Use the native ERNIE4.5 MoE implementation at the pinned runtime revision. Force the exact Baidu User:/Assistant: chat template through Jinja, with one CPU slot and 4,096 total context tokens. Reserve 128 tokens for output. No thinking parser, tool executor or accelerator is configured.

   CommandCopy command

   cd "$HOME/ernie45-hosting"
   export ERNIE45_SERVICE_KEY="$(cat .api-key)"
   ./llama.cpp/build/bin/llama-server -m models/ERNIE-4.5-21B-A3B-PT-Q4_K_M.gguf --alias ernie45-21ba3b --host 127.0.0.1 --port 8080 --api-key "$ERNIE45_SERVICE_KEY" --device none -ngl 0 -t 4 -c 4096 -np 1 --cache-type-k f16 --cache-type-v f16 --jinja --chat-template-file ernie-chat.jinja --reasoning off --temp 0.8 --top-k 0 --top-p 0.8 --min-p 0.0 --repeat-penalty 1.0 --frequency-penalty 0.0 --presence-penalty 0.0

   Expected result: Model initializes and listener starts. Pass health and authenticated chat checks before relying on the service. Command-line credentials may be visible to privileged/local users; trusted host required.

Verify the setup

1. 1. Check readiness and authenticated text generation

   Run checks from a second terminal on the server. Health is public on loopback. Check the authenticated chat’s non-empty final content against the supplied passage and ensure finish_reason is not length; no exact wording is guaranteed. This is a short functional summarization check, not a quality benchmark.

   CommandCopy command

   cd "$HOME/ernie45-hosting"
   export ERNIE45_SERVICE_KEY="$(cat .api-key)"
   curl --fail http://127.0.0.1:8080/health
   curl --fail http://127.0.0.1:8080/v1/chat/completions -H "Authorization: Bearer $ERNIE45_SERVICE_KEY" -H 'Content-Type: application/json' -d '{"model":"ernie45-21ba3b","messages":[{"role":"user","content":"Summarize this supplied passage in one sentence without adding facts: The community library opens at 9am on weekdays and closes at 6pm. On Saturday it opens at 10am and closes at 4pm. It is closed on Sunday."}],"max_tokens":128,"temperature":0.8,"top_k":0,"top_p":0.8,"min_p":0.0,"repeat_penalty":1.0}'
   curl -o /dev/null -s -w '%{http_code}\n' http://127.0.0.1:8080/v1/chat/completions -H 'Content-Type: application/json' -d '{"model":"ernie45-21ba3b","messages":[{"role":"user","content":"Hello"}]}'

   Expected result: Health HTTP 200 when ready; authorized response contains non-empty choices[0].message.content accurately summarizing the supplied opening hours, with finish_reason not length; unauthorized chat returns HTTP 401. Inspect facts yourself; generated text can be wrong.

Performance evidence

No compatible measurement available. Documented support and estimated memory do not establish response speed.

Limits and troubleshooting

• Documented architecture support only. Exact full-process memory, fit, latency, throughput and answer quality are unknown. No model execution occurred.
• This is the text post-trained non-Thinking checkpoint. Publisher overview lists 21B total / 3B active per token. Training-family multimodal discussion does not make this text model a vision guide.
• The source config supports 131,072 positions; this recipe deliberately configures 4,096 combined prompt/output tokens, one sequence and a 128-token validation output cap.
• Baidu generation_config gives temperature 0.8, top_p 0.8, repetition penalty 1.0 and zero frequency/presence penalties. Additional top-k/min-p filtering is disabled explicitly. No sibling sampler or speed is transferred.
• Native template emits begin-of-sentence, User:, Assistant: and end-of-sentence tokens. The pinned original template is explicitly loaded; do not replace it with a generic or Thinking template.
• 32 GiB RAM and 40 GiB disk are planning allocations, not measured requirements. Full experts and runtime state remain relevant despite 3B active computation.
• System packages and build dependencies resolve at installation. Save versions and build logs on execution; build GGML_NATIVE on the same CPU used for inference.
• The quantizer is Unsloth; its own derivative card and metadata link the Baidu base and Apache 2.0 license. Quantization quality has not been measured here.
• Loopback and a private service key require a trusted host. Shared/public production access needs separately designed isolation, TLS and operations.
• Whole-model coverage is not final; this pair documents one useful private CPU text configuration.

Download or checksum failure

Check exact pinned Unsloth GGUF and Baidu template URLs, filename case and both hashes. Do not serve mismatched bytes.

Unsupported architecture or build failure

Use exact llama.cpp commit 620d2225cc436ab0b001e3200e590e3b286ef9ab with native ERNIE4.5 MoE support. Do not silently substitute an older package.

Out of memory or context overflow

Stop the service and inspect host memory. Shorten prompt/output within 4,096 total tokens or explicitly change allocation. Full fit remains unknown.

HTTP 401 or connection refused

Check foreground server startup, loopback port 8080 and matching Bearer service key. Check the operator SSH tunnel; do not expose API port publicly.

Poor or truncated summary

Confirm non-Thinking PT-derived GGUF, exact publisher template and sampler. Inspect finish_reason and supplied facts. Increase output cap only within total context and record the changed workload; smoke checks do not establish reliable quality.

Stop the model

1. 1. Stop the service

   Press Ctrl+C in server terminal. Unset key in each shell. Close the SSH tunnel with Ctrl+C. Stopping the process alone does not stop EC2 billing.

   CommandCopy command

   unset ERNIE45_SERVICE_KEY

   Expected result: Server stops and localhost:8080 stops serving.

Uninstall

1. 1. Remove dedicated model, build and credential files

   Back up wanted logs/files first. Remove only this walkthrough directory after confirming its path. Shared apt packages can remain; do not remove other workloads.

   CommandCopy command

   rm -r -- "$HOME/ernie45-hosting"

   Expected result: Dedicated files and local service credential removed; shared system dependencies remain.

Shut down cloud resources

1. 1. End cloud billing resources

   In EC2 us-east-1, stop for later reuse or terminate this exact VM. Review Delete on termination for its root volume. Delete unwanted detached EBS volumes, snapshots and reserved/Elastic/public IP resources. Stopped VM storage and retained resources may still bill. Check Bills/Cost Explorer later for final usage.

   Expected result: Exact instance is stopped/terminated as intended; no unwanted attached/retained billed resources remain. Instance compute stops only after its lifecycle transition, not when llama-server stops.

Related guides

• https://opentools.ai/llms/ernie-45-21b-a3b