Hermes 3 70B Instruct

Exact model and requirements

Artifact:

Nous Research / Hermes-3-Llama-3.1-70B Q4_K_M · checkpoint NousResearch/Hermes-3-Llama-3.1-70B-GGUF

Publisher revision:

2426f0fa281115286ea90967a480666fa502bade

Artifact digest:

sha256:955c2f42caade4278f3c9dbffa32bb74572652b20e49e5340e782de3585bbe3f

Download size:

39.6 GiB

Memory requirement (unknown):

Unknown — No exact CPU peak for this dense 70B Q4_K_M artifact and 4K workload.

System RAM / disk:

Unknown / Unknown

Placement:

single device. One CPU host; no tensor/pipeline sharding. CPU-only execution; --device none and zero GPU layers. No disk or accelerator offload.

KV cache:

F16 K and V attention cache

Workload:

4,096 token context; Unknown; Unknown; concurrency 1; batch 1; cache unknown; repetitions Unknown.

Tokenizer:

Exact Nous GGUF tokenizer with separately extracted native default ChatML template · 2426f0fa281115286ea90967a480666fa502bade; native template 1fc86da0ce9cdb14cd775ad270bc7d1b4bf70ede; derived sha256 9ff2eb30a1d3e937a88108d74b1ab940891be58aa742be4e8c0ce1c1ad90588e

Sampling / reasoning:

Exact native generation_config supplies temperature 0.6 and top_p 0.9. Author disables extra top_k/min_p filters with 0, sets repeat_penalty 1 and frequency/presence penalties 0. Output cap 512. Plain native default ChatML chat template. No thinking/keep_cots kwargs, forced reasoning tags, tool parser or guaranteed no-deliberation claim.

Runtime prerequisites:

Ubuntu 24.04 x86_64; See prerequisites above.

Official artifact · License

Both native and Nous Research GGUF metadata declare llama3, while the native card identifies Llama 3.1 as the base. The native root LICENSE is absent (404). Retain this discrepancy: this guide does not replace the publisher label or claim the two licenses equivalent. Read and comply with the Meta Llama 3 and Llama 3.1 Community License Agreements and both incorporated acceptable-use policies, including applicable attribution/notice, Built with branding, redistribution/model-naming, prohibited-use, other-model training restrictions and 700-million monthly-active-user commercial-license conditions. Both provide conditional use grants; this is restricted open weights, not Apache or unrestricted open source. Stop if your use cannot satisfy the applicable terms. No legal-clearance claim is made.

• Native tokenizer_config.json with named default ChatML template · 0 GiB · revision 1fc86da0ce9cdb14cd775ad270bc7d1b4bf70ede

• The full 42,520,392,992-byte dense-model weights remain relevant; no MoE active-parameter discount. File bytes are not total working memory.
• 64 GiB RAM, 80 GiB disk and eight inference threads are planning allocations. Cache, compute buffers, runtime and OS add resources; full fit and performance remain unknown.

Install the runtime

1. 1. Build the pinned CPU server

   Install build dependencies and compile the exact official source revision. These packages are system shared dependencies; no CUDA/GPU driver is required. GGML_NATIVE=ON selects -march=native for this GNU/Linux x86 build, compiling for the host CPU. Build and run on the same machine; no AVX-512 requirement is imposed on the i5-12400.

   CommandCopy command

   sudo apt-get update
   sudo apt-get install -y build-essential cmake git curl ca-certificates openssl python3
   mkdir -p "$HOME/hermes3-70b-hosting/models"
   cd "$HOME/hermes3-70b-hosting"
   git clone https://github.com/ggml-org/llama.cpp.git
   cd llama.cpp
   git checkout --detach 620d2225cc436ab0b001e3200e590e3b286ef9ab
   cmake -S . -B build -DCMAKE_BUILD_TYPE=Release -DGGML_METAL=OFF -DGGML_NATIVE=ON
   cmake --build build --config Release -j 4
   ./build/bin/llama-server --version

   Expected result: Build succeeds and server version identifies pinned source. Stop if checkout or build fails.

Download the model

1. 1. Download exact Nous Research GGUF and native tokenizer configuration

   The GGUF is published by Nous Research, with independently pinned native tokenizer JSON. Exact upstream conversion commit is unknown. Verify both files before deriving the default template; the native standalone template path returned 404.

   CommandCopy command

   cd "$HOME/hermes3-70b-hosting"
   curl --fail --location --output models/Hermes-3-Llama-3.1-70B.Q4_K_M.gguf 'https://huggingface.co/NousResearch/Hermes-3-Llama-3.1-70B-GGUF/resolve/2426f0fa281115286ea90967a480666fa502bade/Hermes-3-Llama-3.1-70B.Q4_K_M.gguf'
   curl --fail --location --output models/tokenizer_config.json 'https://huggingface.co/NousResearch/Hermes-3-Llama-3.1-70B/raw/1fc86da0ce9cdb14cd775ad270bc7d1b4bf70ede/tokenizer_config.json'
   sha256sum --check <<'SHA256'
   955c2f42caade4278f3c9dbffa32bb74572652b20e49e5340e782de3585bbe3f  models/Hermes-3-Llama-3.1-70B.Q4_K_M.gguf
   53c6d49282876ffbbef01a9f34968a03187ddb33ea2d596f58dabba247761ea2  models/tokenizer_config.json
   SHA256

   Expected result: Both files report OK. GGUF is 42,520,392,992 bytes; tokenizer JSON is 55,808 bytes. Stop on any mismatch.

Configure

1. 1. Create a private service credential

   Generate this only on your own host when following the guide. It is a local service password, not an OpenAI/provider credential. Keep it private and never paste it into tickets. Create it only when absent; preserve an existing key so a repeated setup does not silently rotate access. It is not created during this documentation run.

   CommandCopy command

   cd "$HOME/hermes3-70b-hosting"
   umask 077
   if [ ! -e .api-key ]; then
     openssl rand -hex 32 > .api-key
   fi
   test -s .api-key || { echo 'Existing .api-key is empty; restore it before continuing.'; exit 1; }
   chmod 600 .api-key

   Expected result: A non-empty private .api-key file exists; existing key preserved. No secret is printed.

2. 2. Extract the exact named default ChatML template

   The native JSON chat_template value is a list with separate default and tool_use entries. Select exactly one default entry, preserve all UTF-8 bytes and add no newline. This is a documented derivation, not a standalone publisher template URL.

   CommandCopy command

   cd "$HOME/hermes3-70b-hosting"
   python3 - <<'PYTEMPLATE'
   import pathlib,json,hashlib
   config=json.loads(pathlib.Path('models/tokenizer_config.json').read_text(encoding='utf-8'))
   entries=config['chat_template']
   assert isinstance(entries,list)
   selected=[entry['template'] for entry in entries if entry.get('name')=='default']
   assert len(selected)==1 and isinstance(selected[0],str)
   raw=selected[0].encode('utf-8')
   assert len(raw)==341 and hashlib.sha256(raw).hexdigest()=='9ff2eb30a1d3e937a88108d74b1ab940891be58aa742be4e8c0ce1c1ad90588e'
   pathlib.Path('models/hermes-chat.jinja').write_bytes(raw)
   assert pathlib.Path('models/hermes-chat.jinja').read_bytes()==selected[0].encode('utf-8')
   print('Default native ChatML template extracted and verified')
   PYTEMPLATE

   Expected result: Exact 341-byte default template verified. It adds a helpful-assistant system prompt when absent, formats ChatML turns and appends an assistant generation prompt.

Start the model

1. 1. Start one bounded CPU text-chat sequence

   Use the exact native default ChatML template and one CPU sequence. Total 4,096 tokens includes prompt, template and up to 512 output tokens. This is a short functional workload below the native 131,072 positions, not a fit or long-context claim. No Hermes 4 thinking kwargs are supplied.

   CommandCopy command

   cd "$HOME/hermes3-70b-hosting"
   export HERMES3_SERVICE_KEY="$(cat .api-key)"
   ./llama.cpp/build/bin/llama-server -m models/Hermes-3-Llama-3.1-70B.Q4_K_M.gguf --alias hermes3-70b --host 127.0.0.1 --port 8080 --api-key "$HERMES3_SERVICE_KEY" --device none -ngl 0 -t 8 -c 4096 -np 1 --cache-type-k f16 --cache-type-v f16 --jinja --chat-template-file models/hermes-chat.jinja --skip-chat-parsing --reasoning-format none --temp 0.6 --top-k 0 --top-p 0.9 --min-p 0 --repeat-penalty 1 --frequency-penalty 0 --presence-penalty 0

   Expected result: Model initializes and loopback listener starts. Stop on tokenizer/architecture/template errors. The service key in process arguments can be visible to trusted local or privileged users.

Verify the setup

1. 1. Check readiness, authenticated chat and missing-key rejection

   Run in a second terminal on the same host. Require a nonempty complete final answer giving 36 counters and finish_reason other than length. No exact generated wording or correctness is guaranteed; these are reader pass criteria, not observed results.

   CommandCopy command

   cd "$HOME/hermes3-70b-hosting"
   export HERMES3_SERVICE_KEY="$(cat .api-key)"
   curl --fail http://127.0.0.1:8080/health
   curl --fail http://127.0.0.1:8080/v1/chat/completions -H "Authorization: Bearer $HERMES3_SERVICE_KEY" -H 'Content-Type: application/json' --data-binary '{"model":"hermes3-70b","messages":[{"role":"user","content":"A box has 12 red counters, 18 blue counters and 6 green counters. Give the total number of counters in one short sentence."}],"max_tokens":512,"temperature":0.6,"top_k":0,"top_p":0.9,"min_p":0,"repeat_penalty":1,"frequency_penalty":0,"presence_penalty":0,"stop":["<|im_end|>","<|end_of_text|>"]}' --output chat-response.json
   cat chat-response.json
   curl -s -o /dev/null -w '%{http_code}\n' http://127.0.0.1:8080/v1/chat/completions -H 'Content-Type: application/json' --data-binary '{"model":"hermes3-70b","messages":[{"role":"user","content":"Hello"}]}'

   Expected result: Ready health HTTP 200, authenticated nonempty complete answer 36 counters with finish_reason not length, and missing-key HTTP 401. These are reader pass criteria, not observed results.

Performance evidence

No compatible measurement available. Documented support and estimated memory do not establish response speed.

Limits and troubleshooting

• Documented configuration only. No weights, install/build, inference, peak-memory, speed or quality measurements were performed.
• Dense Llama 3.1 architecture has 80 layers, hidden size 8,192, 64 attention heads, eight KV heads and Llama3 rope scaling. 131,072 native positions are distinct from this 4,096 total context configuration.
• Native template is the named default entry in tokenizer JSON, extracted verbatim and hash verified. The separate tool_use entry is not used. Embedded GGUF template equality and exact source conversion revision are unknown.
• This is Hermes 3 ChatML text chat, not Hermes 4 hybrid-mode configuration. No thinking or keep_cots switches, forced reasoning tags or no-deliberation guarantee is configured.
• The exact native generation_config supplies .6 temperature and .9 top_p. Extra top-k/min-p filters disabled and repetition/frequency/presence controls are author choices; no Hermes 4 preset transferred.
• <|im_end|> is native EOS token 128039; <|end_of_text|> is native special token 128001. Both stop strings are explicitly included for the smoke request.
• A complete accurate final answer and nonlength finish are required. The 512-token cap may truncate longer requests; changing context/output is a changed workload requiring validation.
• Both native and Nous Research GGUF metadata declare llama3, while the native card identifies Llama 3.1 as the base. The native root LICENSE is absent (404). Retain this discrepancy: this guide does not replace the publisher label or claim the two licenses equivalent. Read and comply with the Meta Llama 3 and Llama 3.1 Community License Agreements and both incorporated acceptable-use policies, including applicable attribution/notice, Built with branding, redistribution/model-naming, prohibited-use, other-model training restrictions and 700-million monthly-active-user commercial-license conditions. Both provide conditional use grants; this is restricted open weights, not Apache or unrestricted open source. Stop if your use cannot satisfy the applicable terms. No legal-clearance claim is made.
• 64 GiB RAM, 80 GiB disk and eight CPU threads are planning allocations, not measured minima. Full process cache/buffers/OS state must fit; file size alone does not establish that.
• The native card documents function calling, structured JSON, roleplaying and long-context use. These useful scopes, other operating systems/backends and accelerator deployments require separate source-backed recipes; this pair does not exhaust model coverage.
• Build on the CPU that will run GGML_NATIVE. Save tool versions/build logs; package dependencies resolve at installation. Loopback/private key assume a trusted host, with public serving requiring separate isolation/TLS/operations.

Checksum or template extraction error

Stop on mismatches. Use exact Nous GGUF and native tokenizer JSON pins; select the unique default entry and verify its derived hash. Do not use a generic Llama or Hermes 4 template.

Poor or truncated answer

Inspect complete final content and finish_reason. The smoke check fails on length truncation or incorrect total. Any larger context/output allocation needs its changed workload recorded.

Memory pressure

Stop and inspect actual resource use. Planning allocation is not measured fit. Change resources or quantization explicitly; do not infer working memory from file bytes.

HTTP 401 or connection refused

Check foreground model startup, loopback 8080 and matching private service key. Run the client on this same desktop.

Stop the model

1. 1. Stop the service

   Press Ctrl+C in server terminal. Unset key in each shell. Confirm this desktop localhost:8080 no longer serves the model.

   CommandCopy command

   unset HERMES3_SERVICE_KEY

   Expected result: Server stops and localhost:8080 stops serving.

Uninstall

1. 1. Remove dedicated model, build and credential files

   Back up wanted logs/files first. Remove only this walkthrough directory after confirming its path. Shared apt packages can remain; do not remove other workloads.

   CommandCopy command

   rm -r -- "$HOME/hermes3-70b-hosting"

   Expected result: Dedicated files and local service credential removed; shared system dependencies remain.

Related guides

• https://opentools.ai/llms/hermes-3-70b-instruct

Exact model and requirements

Artifact:

Nous Research / Hermes-3-Llama-3.1-70B Q4_K_M · checkpoint NousResearch/Hermes-3-Llama-3.1-70B-GGUF

Publisher revision:

2426f0fa281115286ea90967a480666fa502bade

Artifact digest:

sha256:955c2f42caade4278f3c9dbffa32bb74572652b20e49e5340e782de3585bbe3f

Download size:

39.6 GiB

Memory requirement (unknown):

Unknown — No exact CPU peak for this dense 70B Q4_K_M artifact and 4K workload.

System RAM / disk:

Unknown / Unknown

Placement:

single device. One CPU host; no tensor/pipeline sharding. CPU-only execution; --device none and zero GPU layers. No disk or accelerator offload.

KV cache:

F16 K and V attention cache

Workload:

4,096 token context; Unknown; Unknown; concurrency 1; batch 1; cache unknown; repetitions Unknown.

Tokenizer:

Exact Nous GGUF tokenizer with separately extracted native default ChatML template · 2426f0fa281115286ea90967a480666fa502bade; native template 1fc86da0ce9cdb14cd775ad270bc7d1b4bf70ede; derived sha256 9ff2eb30a1d3e937a88108d74b1ab940891be58aa742be4e8c0ce1c1ad90588e

Sampling / reasoning:

Exact native generation_config supplies temperature 0.6 and top_p 0.9. Author disables extra top_k/min_p filters with 0, sets repeat_penalty 1 and frequency/presence penalties 0. Output cap 512. Plain native default ChatML chat template. No thinking/keep_cots kwargs, forced reasoning tags, tool parser or guaranteed no-deliberation claim.

Runtime prerequisites:

Ubuntu 24.04 x86_64; See prerequisites above.

Official artifact · License

Both native and Nous Research GGUF metadata declare llama3, while the native card identifies Llama 3.1 as the base. The native root LICENSE is absent (404). Retain this discrepancy: this guide does not replace the publisher label or claim the two licenses equivalent. Read and comply with the Meta Llama 3 and Llama 3.1 Community License Agreements and both incorporated acceptable-use policies, including applicable attribution/notice, Built with branding, redistribution/model-naming, prohibited-use, other-model training restrictions and 700-million monthly-active-user commercial-license conditions. Both provide conditional use grants; this is restricted open weights, not Apache or unrestricted open source. Stop if your use cannot satisfy the applicable terms. No legal-clearance claim is made.

• Native tokenizer_config.json with named default ChatML template · 0 GiB · revision 1fc86da0ce9cdb14cd775ad270bc7d1b4bf70ede

• The full 42,520,392,992-byte dense-model weights remain relevant; no MoE active-parameter discount. File bytes are not total working memory.
• 64 GiB RAM, 80 GiB disk and eight inference threads are planning allocations. Cache, compute buffers, runtime and OS add resources; full fit and performance remain unknown.

Install the runtime

1. 1. Build the pinned CPU server

   Install build dependencies and compile the exact official source revision. These packages are system shared dependencies; no CUDA/GPU driver is required. GGML_NATIVE=ON selects -march=native for this GNU/Linux x86 build, compiling for the host CPU. Build and run on the same machine; the binary uses the instruction set exposed by this r7i.2xlarge guest. Do not copy it to a different instance family without rebuilding.

   CommandCopy command

   sudo apt-get update
   sudo apt-get install -y build-essential cmake git curl ca-certificates openssl python3
   mkdir -p "$HOME/hermes3-70b-hosting/models"
   cd "$HOME/hermes3-70b-hosting"
   git clone https://github.com/ggml-org/llama.cpp.git
   cd llama.cpp
   git checkout --detach 620d2225cc436ab0b001e3200e590e3b286ef9ab
   cmake -S . -B build -DCMAKE_BUILD_TYPE=Release -DGGML_METAL=OFF -DGGML_NATIVE=ON
   cmake --build build --config Release -j 4
   ./build/bin/llama-server --version

   Expected result: Build succeeds and server version identifies pinned source. Stop if checkout or build fails.

Download the model

1. 1. Download exact Nous Research GGUF and native tokenizer configuration

   The GGUF is published by Nous Research, with independently pinned native tokenizer JSON. Exact upstream conversion commit is unknown. Verify both files before deriving the default template; the native standalone template path returned 404.

   CommandCopy command

   cd "$HOME/hermes3-70b-hosting"
   curl --fail --location --output models/Hermes-3-Llama-3.1-70B.Q4_K_M.gguf 'https://huggingface.co/NousResearch/Hermes-3-Llama-3.1-70B-GGUF/resolve/2426f0fa281115286ea90967a480666fa502bade/Hermes-3-Llama-3.1-70B.Q4_K_M.gguf'
   curl --fail --location --output models/tokenizer_config.json 'https://huggingface.co/NousResearch/Hermes-3-Llama-3.1-70B/raw/1fc86da0ce9cdb14cd775ad270bc7d1b4bf70ede/tokenizer_config.json'
   sha256sum --check <<'SHA256'
   955c2f42caade4278f3c9dbffa32bb74572652b20e49e5340e782de3585bbe3f  models/Hermes-3-Llama-3.1-70B.Q4_K_M.gguf
   53c6d49282876ffbbef01a9f34968a03187ddb33ea2d596f58dabba247761ea2  models/tokenizer_config.json
   SHA256

   Expected result: Both files report OK. GGUF is 42,520,392,992 bytes; tokenizer JSON is 55,808 bytes. Stop on any mismatch.

Configure

1. 1. Create a private service credential

   Generate this only on your own host when following the guide. It is a local service password, not an OpenAI/provider credential. Keep it private and never paste it into tickets. Create it only when absent; preserve an existing key so a repeated setup does not silently rotate access. It is not created during this documentation run.

   CommandCopy command

   cd "$HOME/hermes3-70b-hosting"
   umask 077
   if [ ! -e .api-key ]; then
     openssl rand -hex 32 > .api-key
   fi
   test -s .api-key || { echo 'Existing .api-key is empty; restore it before continuing.'; exit 1; }
   chmod 600 .api-key

   Expected result: A non-empty private .api-key file exists; existing key preserved. No secret is printed.

2. 2. Extract the exact named default ChatML template

   The native JSON chat_template value is a list with separate default and tool_use entries. Select exactly one default entry, preserve all UTF-8 bytes and add no newline. This is a documented derivation, not a standalone publisher template URL.

   CommandCopy command

   cd "$HOME/hermes3-70b-hosting"
   python3 - <<'PYTEMPLATE'
   import pathlib,json,hashlib
   config=json.loads(pathlib.Path('models/tokenizer_config.json').read_text(encoding='utf-8'))
   entries=config['chat_template']
   assert isinstance(entries,list)
   selected=[entry['template'] for entry in entries if entry.get('name')=='default']
   assert len(selected)==1 and isinstance(selected[0],str)
   raw=selected[0].encode('utf-8')
   assert len(raw)==341 and hashlib.sha256(raw).hexdigest()=='9ff2eb30a1d3e937a88108d74b1ab940891be58aa742be4e8c0ce1c1ad90588e'
   pathlib.Path('models/hermes-chat.jinja').write_bytes(raw)
   assert pathlib.Path('models/hermes-chat.jinja').read_bytes()==selected[0].encode('utf-8')
   print('Default native ChatML template extracted and verified')
   PYTEMPLATE

   Expected result: Exact 341-byte default template verified. It adds a helpful-assistant system prompt when absent, formats ChatML turns and appends an assistant generation prompt.

Start the model

1. 1. Start one bounded CPU text-chat sequence

   Use the exact native default ChatML template and one CPU sequence. Total 4,096 tokens includes prompt, template and up to 512 output tokens. This is a short functional workload below the native 131,072 positions, not a fit or long-context claim. No Hermes 4 thinking kwargs are supplied.

   CommandCopy command

   cd "$HOME/hermes3-70b-hosting"
   export HERMES3_SERVICE_KEY="$(cat .api-key)"
   ./llama.cpp/build/bin/llama-server -m models/Hermes-3-Llama-3.1-70B.Q4_K_M.gguf --alias hermes3-70b --host 127.0.0.1 --port 8080 --api-key "$HERMES3_SERVICE_KEY" --device none -ngl 0 -t 8 -c 4096 -np 1 --cache-type-k f16 --cache-type-v f16 --jinja --chat-template-file models/hermes-chat.jinja --skip-chat-parsing --reasoning-format none --temp 0.6 --top-k 0 --top-p 0.9 --min-p 0 --repeat-penalty 1 --frequency-penalty 0 --presence-penalty 0

   Expected result: Model initializes and loopback listener starts. Stop on tokenizer/architecture/template errors. The service key in process arguments can be visible to trusted local or privileged users.

Verify the setup

1. 1. Check readiness, authenticated chat and missing-key rejection

   Run in a second terminal on the same host. Require a nonempty complete final answer giving 36 counters and finish_reason other than length. No exact generated wording or correctness is guaranteed; these are reader pass criteria, not observed results.

   CommandCopy command

   cd "$HOME/hermes3-70b-hosting"
   export HERMES3_SERVICE_KEY="$(cat .api-key)"
   curl --fail http://127.0.0.1:8080/health
   curl --fail http://127.0.0.1:8080/v1/chat/completions -H "Authorization: Bearer $HERMES3_SERVICE_KEY" -H 'Content-Type: application/json' --data-binary '{"model":"hermes3-70b","messages":[{"role":"user","content":"A box has 12 red counters, 18 blue counters and 6 green counters. Give the total number of counters in one short sentence."}],"max_tokens":512,"temperature":0.6,"top_k":0,"top_p":0.9,"min_p":0,"repeat_penalty":1,"frequency_penalty":0,"presence_penalty":0,"stop":["<|im_end|>","<|end_of_text|>"]}' --output chat-response.json
   cat chat-response.json
   curl -s -o /dev/null -w '%{http_code}\n' http://127.0.0.1:8080/v1/chat/completions -H 'Content-Type: application/json' --data-binary '{"model":"hermes3-70b","messages":[{"role":"user","content":"Hello"}]}'

   Expected result: Ready health HTTP 200, authenticated nonempty complete answer 36 counters with finish_reason not length, and missing-key HTTP 401. These are reader pass criteria, not observed results.

Performance evidence

No compatible measurement available. Documented support and estimated memory do not establish response speed.

Limits and troubleshooting

• Documented configuration only. No weights, install/build, inference, peak-memory, speed or quality measurements were performed.
• Dense Llama 3.1 architecture has 80 layers, hidden size 8,192, 64 attention heads, eight KV heads and Llama3 rope scaling. 131,072 native positions are distinct from this 4,096 total context configuration.
• Native template is the named default entry in tokenizer JSON, extracted verbatim and hash verified. The separate tool_use entry is not used. Embedded GGUF template equality and exact source conversion revision are unknown.
• This is Hermes 3 ChatML text chat, not Hermes 4 hybrid-mode configuration. No thinking or keep_cots switches, forced reasoning tags or no-deliberation guarantee is configured.
• The exact native generation_config supplies .6 temperature and .9 top_p. Extra top-k/min-p filters disabled and repetition/frequency/presence controls are author choices; no Hermes 4 preset transferred.
• <|im_end|> is native EOS token 128039; <|end_of_text|> is native special token 128001. Both stop strings are explicitly included for the smoke request.
• A complete accurate final answer and nonlength finish are required. The 512-token cap may truncate longer requests; changing context/output is a changed workload requiring validation.
• Both native and Nous Research GGUF metadata declare llama3, while the native card identifies Llama 3.1 as the base. The native root LICENSE is absent (404). Retain this discrepancy: this guide does not replace the publisher label or claim the two licenses equivalent. Read and comply with the Meta Llama 3 and Llama 3.1 Community License Agreements and both incorporated acceptable-use policies, including applicable attribution/notice, Built with branding, redistribution/model-naming, prohibited-use, other-model training restrictions and 700-million monthly-active-user commercial-license conditions. Both provide conditional use grants; this is restricted open weights, not Apache or unrestricted open source. Stop if your use cannot satisfy the applicable terms. No legal-clearance claim is made.
• 64 GiB RAM, 80 GiB disk and eight CPU threads are planning allocations, not measured minima. Full process cache/buffers/OS state must fit; file size alone does not establish that.
• The native card documents function calling, structured JSON, roleplaying and long-context use. These useful scopes, other operating systems/backends and accelerator deployments require separate source-backed recipes; this pair does not exhaust model coverage.
• Build on the CPU that will run GGML_NATIVE. Save tool versions/build logs; package dependencies resolve at installation. Loopback/private key assume a trusted host, with public serving requiring separate isolation/TLS/operations.

Checksum or template extraction error

Stop on mismatches. Use exact Nous GGUF and native tokenizer JSON pins; select the unique default entry and verify its derived hash. Do not use a generic Llama or Hermes 4 template.

Poor or truncated answer

Inspect complete final content and finish_reason. The smoke check fails on length truncation or incorrect total. Any larger context/output allocation needs its changed workload recorded.

Memory pressure

Stop and inspect actual resource use. Planning allocation is not measured fit. Change resources or quantization explicitly; do not infer working memory from file bytes.

HTTP 401 or connection refused

Check foreground model startup, loopback 8080 and matching private service key. Check SSH forwarding to the intended VM; keep public API ingress closed.

Stop the model

1. 1. Stop the service

   Press Ctrl+C in server terminal. Unset key in each shell. Close the SSH tunnel with Ctrl+C. Stopping the process alone does not stop EC2 billing.

   CommandCopy command

   unset HERMES3_SERVICE_KEY

   Expected result: Server stops and localhost:8080 stops serving.

Uninstall

1. 1. Remove dedicated model, build and credential files

   Back up wanted logs/files first. Remove only this walkthrough directory after confirming its path. Shared apt packages can remain; do not remove other workloads.

   CommandCopy command

   rm -r -- "$HOME/hermes3-70b-hosting"

   Expected result: Dedicated files and local service credential removed; shared system dependencies remain.

Shut down cloud resources

1. 1. End cloud billing resources

   In EC2 us-east-1, stop for later reuse or terminate this exact VM. Review Delete on termination for its root volume. Delete unwanted detached EBS volumes, snapshots and reserved/Elastic/public IP resources. Stopped VM storage and retained resources may still bill. Check Bills/Cost Explorer later for final usage.

   Expected result: Exact instance is stopped/terminated as intended; no unwanted attached/retained billed resources remain. Instance compute stops only after its lifecycle transition, not when llama-server stops.

Related guides

• https://opentools.ai/llms/hermes-3-70b-instruct