Local hardwareDocumented setupSources checked Oct 16, 2026
Run complete Command R+ (08-2024) Q4_K_M on a local CPU
Load both exact Bartowski Q4_K_M shards for the complete Command R+ (08-2024) model and its independently pinned native ordinary-chat template. The private non-commercial CPU walkthrough uses one sequence, 4,096 total context tokens and a 512-token output cap. The 128 GiB host allocation is planning headroom; full fit, speed and answer quality remain unmeasured.
- Hardware
- Owned Intel Core i5-12400 host, 8 inference threads, 128 GiB RAM
- Runtime
- llama.cpp v0.7.0 (620d2225; full pin in source URL) · CPU
- Model format
- GGUF · Complete two-shard Q4_K_M mixed GGUF; no runtime quantization or native FP32 claim · Complete two-shard Q4_K_M mixed GGUF quantization; not native FP32
- Workload
- 4,096 token context · 1 concurrent
Based on source documentation; this setup has not been executed by OpenTools. Memory: unknown. No compatible measurement available.
Read the complete setup guide
Before you start
- Use an owned Ubuntu 24.04 x86_64 Intel CPU host with eight inference threads and an actually available 128 GiB system RAM / 160 GiB free disk allocation. Verify the motherboard/DIMM configuration and actual available capacity before proceeding; the referenced CPU vendor specification is not proof that your whole host is configured that way.
- The selected named GGUF and native metadata declare CC BY-NC 4.0. The actual publisher license incorporates the Cohere Labs Acceptable Use Addendum and policy. NonCommercial means not primarily intended for commercial advantage or monetary compensation; private or paid-VM use is not an exemption. Retain required attribution, license/disclaimer/source notices and indicate modifications when sharing; do not impose additional restrictions on licensed rights. The incorporated policy prohibits specified harmful/illegal/privacy/deceptive uses and synthetic data outputs for commercial purposes, including model or product development. Read all full terms and assess your own intended use before downloading. No terms acceptance, commercial permission or legal clearance is performed for you.
- Use the exact complete August 2024 R+ derivative, with both pinned split files in one directory. The frozen R+ canonical name/API is coherent despite its slug shorthand. This is not R, R7B, a partial shard or native FP32 setup.
- Native raw files are gated in the captured access state. Public official metadata includes the exact native template, and the named derivative repository metadata and README are public. A download access error must stop the guide; resolve through the publisher without gate bypass.
- Review this pinned runtime, public source and template before building. Private loopback/key does not grant commercial rights or create a public TLS/multiuser service. Use original non-sensitive non-commercial research text only.
Exact model and requirements
- Artifact:
- Bartowski (named split GGUF derivative); Cohere Labs (native model) / Complete Command R+ (08-2024) Q4_K_M by Bartowski, both shards · checkpoint bartowski/c4ai-command-r-plus-08-2024-GGUF
- Publisher revision:
- 4cb2af9f9be9b2753915e67aa8d658cd031b8a21
- Artifact digest:
- No single publisher digest for the complete two-shard composition. Both individual LFS SHA256 digests are pinned and checked.
- Download size:
- 58.44 GiB
- Memory requirement (unknown):
- Unknown — No complete process peak measured for both Q4_K_M shards, F16 attention cache, scratch, runtime and OS.
- System RAM / disk:
- Unknown / Unknown
- Placement:
- single device. One CPU host with one conversation slot. All model layers run on CPU; no GPU or disk-offload path is configured.
- KV cache:
- F16 K and V attention cache; actual process/cache allocation unmeasured.
- Workload:
- 4,096 token context; Unknown; Unknown; concurrency 1; batch 1; cache unknown; repetitions Unknown.
- Tokenizer:
- Exact named R+ derivative tokenizer and independently pinned native R+ default template · GGUF derivative pin and native template pin; full revisions in download steps
- Sampling / reasoning:
- Temperature 0.3 follows the actual native R+ August model-card example. Top_k 0, top_p 1, min_p 0 and repeat_penalty 1 are explicit author choices. Native END_OF_TURN stop and 512 output cap; no R7B response markers. Ordinary alternating user/assistant conversation uses the exact 1,255-byte native R+ default template. It emits one BOS and assistant CHATBOT prefix without START_RESPONSE. Both complete GGUF shards are required; native loader discovers the second from the first split filename. Tool/RAG/citation workflows and full 128K context remain separately unqualified.
- Runtime prerequisites:
- Ubuntu 24.04 x86_64; See prerequisites above.
The selected named GGUF and native metadata declare CC BY-NC 4.0. The actual publisher license incorporates the Cohere Labs Acceptable Use Addendum and policy. NonCommercial means not primarily intended for commercial advantage or monetary compensation; private or paid-VM use is not an exemption. Retain required attribution, license/disclaimer/source notices and indicate modifications when sharing; do not impose additional restrictions on licensed rights. The incorporated policy prohibits specified harmful/illegal/privacy/deceptive uses and synthetic data outputs for commercial purposes, including model or product development. Read all full terms and assess your own intended use before downloading. No terms acceptance, commercial permission or legal clearance is performed for you.
- c4ai-command-r-plus-08-2024-Q4_K_M-00001-of-00002.gguf · 37.23 GiB · revision 4cb2af9f9be9b2753915e67aa8d658cd031b8a21
- c4ai-command-r-plus-08-2024-Q4_K_M-00002-of-00002.gguf · 21.22 GiB · revision 4cb2af9f9be9b2753915e67aa8d658cd031b8a21
- Both complete Q4_K_M shards total 62,750,608,416 bytes. Native metadata reports 103,810,674,688 F16 parameters and a 104B model-card label. The first shard alone is not the complete model.
- 128 GiB RAM and 160 GiB disk are allocations; local eight threads and cloud sixteen vCPUs are explicit selections. Full fit, cache/loading peak and CPU speed remain unknown. Native FP32 weights would require a separate much larger-host configuration; that path is not claimed.
One-time setup and hardware
- hardware
- Unknown — Existing desktop purchase price and incremental hardware cost depend on the reader; unknown.
- Existing desktop purchase price and incremental hardware cost depend on the reader; unknown. Tax: unknown.
- setup
- Unknown — Initial provisioning/build time and labor cost are unknown.
- Initial provisioning/build time and labor cost are unknown. Tax: unknown.
Recurring costs
- electricity
- Unknown — Actual wall power, electricity rate and powered hours are unknown; no TDP-to-wall-power conversion.
- Actual wall power, electricity rate and powered hours are unknown; no TDP-to-wall-power conversion. Tax: unknown.
- storage
- Unknown — Incremental cost of local storage is unknown. 160 GiB is an author planning allocation.
- Incremental cost of local storage is unknown. 160 GiB is an author planning allocation. Tax: unknown.
- egress
- Unknown — Data transfer price, allowances and traffic volume are unknown.
- Data transfer price, allowances and traffic volume are unknown. Tax: unknown.
- administration
- Unknown — Maintenance labor rate and hours are unknown.
- Maintenance labor rate and hours are unknown. Tax: unknown.
Install the runtime
1. Build the exact CPU runtime
Install host build dependencies and compile the pinned official source. No CUDA driver is required. Build and run natively on this same CPU; save resolved package/tool versions and build logs during actual execution.
Commandset -e sudo apt-get update sudo apt-get install -y build-essential cmake git curl ca-certificates openssl python3 mkdir -p "$HOME/command-rplus0824-hosting/models" cd "$HOME/command-rplus0824-hosting" git clone https://github.com/ggml-org/llama.cpp.git git -C llama.cpp checkout --detach 620d2225cc436ab0b001e3200e590e3b286ef9ab cmake -S llama.cpp -B llama.cpp/build -DCMAKE_BUILD_TYPE=Release -DGGML_METAL=OFF -DGGML_NATIVE=ON cmake --build llama.cpp/build --config Release -j 4 ./llama.cpp/build/bin/llama-server --versionExpected result: Checkout and build succeed; server version identifies the pinned source. Stop on failure. No inference has been executed by the author.
Download the model
1. Download and verify both complete R+ shards and native template
Read the full terms first. Both exact split files must remain together; each byte count and SHA256 is checked fail-fast before template extraction. Native metadata is checked by canonical model/revision and exact template hash, without binding mutable counters. No shard merging, requantization, conversion or template rewriting is performed.
Commandset -e cd "$HOME/command-rplus0824-hosting" curl --fail --location --retry 0 'https://huggingface.co/bartowski/c4ai-command-r-plus-08-2024-GGUF/resolve/4cb2af9f9be9b2753915e67aa8d658cd031b8a21/c4ai-command-r-plus-08-2024-Q4_K_M/c4ai-command-r-plus-08-2024-Q4_K_M-00001-of-00002.gguf' -o 'models/c4ai-command-r-plus-08-2024-Q4_K_M-00001-of-00002.gguf' echo '8c6a3ecc11e4441cdcc71ffca0c26d49b0f34d0c3aacf38c8b8fb0ce22ebcdc3 models/c4ai-command-r-plus-08-2024-Q4_K_M-00001-of-00002.gguf' | sha256sum --check test "$(stat -c %s 'models/c4ai-command-r-plus-08-2024-Q4_K_M-00001-of-00002.gguf')" = 39970263520 curl --fail --location --retry 0 'https://huggingface.co/bartowski/c4ai-command-r-plus-08-2024-GGUF/resolve/4cb2af9f9be9b2753915e67aa8d658cd031b8a21/c4ai-command-r-plus-08-2024-Q4_K_M/c4ai-command-r-plus-08-2024-Q4_K_M-00002-of-00002.gguf' -o 'models/c4ai-command-r-plus-08-2024-Q4_K_M-00002-of-00002.gguf' echo '5c920b5d6342e8e746019529c056c8c3d30a5b4e4cd087b6620266722a542a2a models/c4ai-command-r-plus-08-2024-Q4_K_M-00002-of-00002.gguf' | sha256sum --check test "$(stat -c %s 'models/c4ai-command-r-plus-08-2024-Q4_K_M-00002-of-00002.gguf')" = 22780344896 curl --fail --location 'https://huggingface.co/api/models/CohereLabs/c4ai-command-r-plus-08-2024/revision/e808c1a2249354ca211c9f08d1338e5039f633f8?blobs=true' -o models/native-metadata.json python3 - <<'PYTEMPLATE' import hashlib,json,pathlib d=json.loads(pathlib.Path('models/native-metadata.json').read_text()) assert d['id']=='CohereLabs/c4ai-command-r-plus-08-2024' and d['sha']=='e808c1a2249354ca211c9f08d1338e5039f633f8' t=next(x['template'] for x in d['config']['tokenizer_config']['chat_template'] if x['name']=='default') b=t.encode('utf-8');assert len(b)==1255 and hashlib.sha256(b).hexdigest()=='469770c7e90472c46cfbd24a081f81e8bc4d99add8e47bc6f61bf44e9cceba88' pathlib.Path('models/command-default.jinja').write_bytes(b) print('Exact native R+ template and both split files verified') PYTEMPLATEExpected result: Both shard byte/checksum checks and exact 1,255-byte native template checks pass. Stop on any mismatch, missing shard or access error.
Configure
1. Create a private service credential
Generate a local server password on your own host when following this guide. Preserve an existing key on repeat setup and never print or share it. This is not an provider API key.
Commandset -e cd "$HOME/command-rplus0824-hosting" umask 077 if [ ! -e .api-key ]; then openssl rand -hex 32 > .api-key fi test -s .api-key || { echo 'Restore the empty service-key file before continuing.'; exit 1; } chmod 600 .api-keyExpected result: A nonempty private .api-key exists with mode 600. Existing key preserved; no secret printed.
Start the model
1. Start the complete private R+ CPU backend
Run in the foreground. Point the native loader at the first exact split filename; its pinned implementation discovers and validates all split files, and a missing split must fail. One slot and 4,096 total context tokens include the complete prompt and up to 512 output tokens. Stop on architecture, shard or template failure.
Commandset -e cd "$HOME/command-rplus0824-hosting" COMMAND_SERVICE_KEY="$(cat .api-key)" test -n "$COMMAND_SERVICE_KEY" export COMMAND_SERVICE_KEY ./llama.cpp/build/bin/llama-server -m models/c4ai-command-r-plus-08-2024-Q4_K_M-00001-of-00002.gguf --alias command-rplus0824 --host 127.0.0.1 --port 8080 --api-key "$COMMAND_SERVICE_KEY" --device none -ngl 0 -t 8 -c 4096 -np 1 --cache-type-k f16 --cache-type-v f16 --jinja --chat-template-file models/command-default.jinja --skip-chat-parsing --reasoning-format none --temp 0.3 --top-k 0 --top-p 1 --min-p 0 --repeat-penalty 1Expected result: The complete native R+ model loads from both split files and serves loopback. No successful model load, fit, startup time or throughput is preclaimed.
Connect to your endpoint
http://127.0.0.1:8080/completion
Trusted desktop loopback only; checks run in a second local terminal.
Access: loopback. Reader-generated private Bearer key protects native template, tokenization and completion routes. Native health is public on loopback. No Cohere provider API credential or paid inference API is used.
Verify the setup
1. Check a complete native non-commercial conversation
Use a second terminal or SSH session on this host. The original library-catalog question and one-sentence follow-up test native conversation formatting and normal completion. Inspect both actual answers; protocol success does not establish correctness or quality. Client timeout does not cancel native generation.
Commandset -e cd "$HOME/command-rplus0824-hosting" cat > chat-client.py <<'PYCLIENT' """Reader-run native Command R+ August 2024 ordinary text conversation check.""" import json,pathlib,urllib.request def check_messages(messages): if not isinstance(messages,list) or not messages or len(messages)%2!=1: raise ValueError('user_ending_conversation_required') for i,m in enumerate(messages): if not isinstance(m,dict) or set(m)!={'role','content'} or m['role']!=('user' if i%2==0 else 'assistant') or not isinstance(m['content'],str) or not m['content'].strip() or len(m['content'].encode())>8192 or '<|' in m['content'] or '<BOS_TOKEN>' in m['content']: raise ValueError('invalid_plain_text_conversation') def run_case(messages,post): check_messages(messages) prompt=post('/apply-template',{'messages':messages})['prompt'] if not isinstance(prompt,str) or not prompt.startswith('<BOS_TOKEN>') or prompt.count('<BOS_TOKEN>')!=1 or not prompt.endswith('<|START_OF_TURN_TOKEN|><|CHATBOT_TOKEN|>'): raise ValueError('native_template_mismatch') ids=post('/tokenize',{'content':prompt,'add_special':False,'parse_special':True})['tokens'] if not isinstance(ids,list) or not ids or any(type(i)!=int or i<0 for i in ids) or len(ids)>3584: raise ValueError('prompt_budget_or_token_shape') result=post('/completion',{'prompt':ids,'n_predict':512,'temperature':0.3,'top_k':0,'top_p':1.0,'min_p':0.0,'repeat_penalty':1.0,'stop':['<|END_OF_TURN_TOKEN|>'],'return_tokens':True,'cache_prompt':False}) if result.get('stop_type') not in ('eos','word') or result.get('truncated') is not False: raise ValueError('unfinished_or_truncated_response') tokens=result.get('tokens');text=result.get('content') if not isinstance(tokens,list) or not 1<=len(tokens)<=512 or any(type(i)!=int or i<0 for i in tokens) or not isinstance(text,str) or not text.strip() or '<|' in text: raise ValueError('response_shape') return {'prompt_tokens':len(ids),'generated_tokens':len(tokens),'response':text} def main(): key=pathlib.Path('.api-key').read_text().strip() def post(path,data): req=urllib.request.Request('http://127.0.0.1:8080'+path,json.dumps(data).encode(),headers={'Authorization':'Bearer '+key,'Content-Type':'application/json'}) with urllib.request.urlopen(req,timeout=600) as response:return json.load(response) first=[{'role':'user','content':'Explain in two short sentences why a library catalog helps a reader find a book.'}] answer=run_case(first,post);print(json.dumps(answer,ensure_ascii=False)) follow=first+[{'role':'assistant','content':answer['response']},{'role':'user','content':'Summarize your explanation in one sentence.'}] print(json.dumps(run_case(follow,post),ensure_ascii=False)) print('Inspect both answers for relevance and coherent follow-up. Syntax and stop checks do not establish correctness or quality.') if __name__=='__main__':main() PYCLIENT python3 chat-client.py code=$(curl -sS -o unauthorized.json -w '%{http_code}' -H 'Content-Type: application/json' --data '{"content":"test"}' http://127.0.0.1:8080/tokenize) test "$code" = 401Expected result: Two actual nonempty responses with counted prompts and normal stops; unauthenticated tokenization returns 401. Human records relevance and coherent follow-up without a forced gold answer.
Performance evidence
No compatible measurement available. Documented support and estimated memory do not establish response speed.
Limits and troubleshooting
- Documented source-qualified setup only; no weight body, build, model load, inference, memory fit, CPU throughput or answer quality was measured.
- The selected named GGUF and native metadata declare CC BY-NC 4.0. The actual publisher license incorporates the Cohere Labs Acceptable Use Addendum and policy. NonCommercial means not primarily intended for commercial advantage or monetary compensation; private or paid-VM use is not an exemption. Retain required attribution, license/disclaimer/source notices and indicate modifications when sharing; do not impose additional restrictions on licensed rights. The incorporated policy prohibits specified harmful/illegal/privacy/deceptive uses and synthetic data outputs for commercial purposes, including model or product development. Read all full terms and assess your own intended use before downloading. No terms acceptance, commercial permission or legal clearance is performed for you.
- This frozen canonical is Command R+ (08-2024), API cohere/command-r-plus-08-2024, distinct from frozen R August and R7B. Native public model card labels 104B, metadata reports 103,810,674,688 F16 parameters; raw gated config dimensions are not invented.
- The complete derivative consists of both exact split GGUF files totaling 62,750,608,416 bytes. Both have individual hashes; no whole-composition digest is invented. Native loader source supports split discovery from the first filename and fails for missing/mismatched splits.
- The native R+ default template is independently pinned at 1,255 UTF-8 bytes. It emits one BOS and assistant CHATBOT prefix, without R7B START_RESPONSE markers. Tool/RAG/citations remain separately unqualified.
- The Bartowski README identifies the native R+ repository and llama.cpp b3634 imatrix quantization, but not its input base commit. Native template and derivative are independent pins requiring actual functional checks, not byte-equivalence claims.
- 4,096 total context and a 512-token output cap are author choices. Native 128K support does not prove fit. F16 cache, runtime/scratch/loading and OS add memory beyond the complete weight file sizes.
- 128 GiB RAM/160 GiB disk are planning allocations. Verify actual local board/DIMM/OS capacity; the exact R7i.4xlarge source states sixteen vCPUs and 128 GiB. No measured fit, peak, speed or quality result exists.
- The private endpoint and paid cloud rental do not exempt NC/AUP terms or create public TLS/multiuser resilience. Native full precision, GPU, long-context and other useful R+ scopes remain open; required total remains null.
- Access error or license uncertainty
- Stop and resolve the exact publisher/derivative access and full terms. Do not bypass a gate, create a new agent credential or assume private hosting or VM payment grants commercial model rights.
- Native template, BOS or architecture mismatch
- Verify exact runtime, complete derivative digest and native template extraction. Do not substitute R7B response markers, a generic template or extra BOS.
- Prompt overflow or partial response
- Shorten the complete conversation within the 3,584-token prompt budget. Limit stop or truncation fails the functional check. A changed context/output budget requires requalification.
- 401 or connection refused
- Check foreground readiness, matching private key and same-host loopback or correctly established SSH tunnel. Keep public port 8080 closed.
- OOM, slow or poor response
- Stop and record actual process memory/resources and answers. Planning allocation, artifact size or normal stop do not prove fit, speed or quality.
Stop the model
1. Stop the CPU backend
Press Ctrl+C in the foreground server terminal. Unset the private key in each shell and close any SSH tunnel. Verify loopback port 8080 closes.
Commandunset COMMAND_SERVICE_KEYExpected result: The backend stops serving loopback port 8080.
Uninstall
1. Remove the dedicated walkthrough files
Back up wanted outputs, confirm the exact path and remove only this dedicated directory. Shared system packages remain.
Commandrm -r -- "$HOME/command-rplus0824-hosting"Expected result: Dedicated complete weights, build, native metadata/template, client, private key and outputs removed.
Sources and verification
Sources checked Oct 16, 2026. Verification source
- https://huggingface.co/api/models/CohereForAI/c4ai-command-r-plus-08-2024?blobs=truepublisher · retrieved Oct 16, 2026 · revision e808c1a22493
- https://docs.cohere.com/docs/command-rpublisher · retrieved Oct 16, 2026
- https://docs.cohere.com/docs/command-r-pluspublisher · retrieved Oct 16, 2026
- https://cohere.com/c4ai-cc-by-nc-licensepublisher · retrieved Oct 16, 2026
- https://creativecommons.org/licenses/by-nc/4.0/legalcode.enpublisher · retrieved Oct 16, 2026
- https://huggingface.co/api/models/bartowski/c4ai-command-r-plus-08-2024-GGUF?blobs=truepublisher · retrieved Oct 16, 2026 · revision 4cb2af9f9be9
- https://docs.cohere.com/docs/cohere-labs-acceptable-use-policypublisher · retrieved Oct 16, 2026
- https://raw.githubusercontent.com/ggml-org/llama.cpp/620d2225cc436ab0b001e3200e590e3b286ef9ab/src/models/command-r.cppruntime maintainer · retrieved Oct 16, 2026 · revision 620d2225
- https://raw.githubusercontent.com/ggml-org/llama.cpp/620d2225cc436ab0b001e3200e590e3b286ef9ab/convert_hf_to_gguf.pyruntime maintainer · retrieved Oct 16, 2026 · revision 620d2225
- https://raw.githubusercontent.com/ggml-org/llama.cpp/620d2225cc436ab0b001e3200e590e3b286ef9ab/conversion/command_r.pyruntime maintainer · retrieved Oct 16, 2026 · revision 620d2225
- https://huggingface.co/bartowski/c4ai-command-r-plus-08-2024-GGUF/raw/4cb2af9f9be9b2753915e67aa8d658cd031b8a21/README.mdpublisher · retrieved Oct 16, 2026 · revision 4cb2af9f9be9
- https://huggingface.co/api/models/CohereLabs/c4ai-command-r-plus-08-2024/revision/e808c1a2249354ca211c9f08d1338e5039f633f8?blobs=truepublisher · retrieved Oct 16, 2026 · revision e808c1a22493
- https://huggingface.co/CohereLabs/c4ai-command-r-plus-08-2024publisher · retrieved Oct 16, 2026 · revision e808c1a22493
- https://huggingface.co/api/models/bartowski/c4ai-command-r-plus-08-2024-GGUF/revision/4cb2af9f9be9b2753915e67aa8d658cd031b8a21?blobs=truepublisher · retrieved Oct 16, 2026 · revision 4cb2af9f9be9
- https://raw.githubusercontent.com/ggml-org/llama.cpp/620d2225cc436ab0b001e3200e590e3b286ef9ab/src/llama-model-loader.cppruntime maintainer · retrieved Oct 16, 2026 · revision 620d2225
- https://opentools.ai/llms/command-r-08-2024opentools · retrieved Oct 16, 2026 · revision august-rplus-native-client-v1
- https://www.intel.com/content/www/us/en/products/sku/134586/intel-core-i512400-processor-18m-cache-up-to-4-40-ghz/specifications.htmlmanufacturer · retrieved Oct 16, 2026
- https://raw.githubusercontent.com/ggml-org/llama.cpp/620d2225cc436ab0b001e3200e590e3b286ef9ab/docs/build.mdruntime maintainer · retrieved Oct 16, 2026 · revision 620d2225cc43
- https://raw.githubusercontent.com/ggml-org/llama.cpp/620d2225cc436ab0b001e3200e590e3b286ef9ab/ggml/src/ggml-cpu/CMakeLists.txtruntime maintainer · retrieved Oct 16, 2026 · revision 620d2225cc43
- https://docs.liquid.ai/lfm/inference/llama-cppruntime maintainer · retrieved Oct 16, 2026
- https://raw.githubusercontent.com/ggml-org/llama.cpp/620d2225cc436ab0b001e3200e590e3b286ef9ab/tools/server/README.mdruntime maintainer · retrieved Oct 16, 2026 · revision 620d2225cc43
- https://raw.githubusercontent.com/ggml-org/llama.cpp/620d2225cc436ab0b001e3200e590e3b286ef9ab/common/arg.cppruntime maintainer · retrieved Oct 16, 2026 · revision 620d2225cc43
- https://api.github.com/repos/ggml-org/llama.cpp/releases/latestruntime maintainer · retrieved Oct 16, 2026