Local hardwareDocumented setupSources checked Oct 16, 2026
Run Command R (08-2024) Q4_K_M on a local CPU
Use the complete named Bartowski Q4_K_M derivative of Command R (08-2024), its independently pinned native ordinary-chat template and CPU llama.cpp. Eight threads, one sequence, 4,096 total context tokens and a 512-token output cap. This bounded non-commercial research conversation checks actual token counts and normal completion; fit, speed and answer quality remain unmeasured.
- Hardware
- Intel Core i5-12400 desktop, 8 inference threads, 64 GiB RAM
- Runtime
- llama.cpp v0.7.0 (620d2225; full pin in source URL) · CPU
- Model format
- GGUF · Complete Q4_K_M mixed GGUF weights; no runtime quantization or native FP32 claim · Complete Bartowski Q4_K_M mixed GGUF quantization; not native FP32
- Workload
- 4,096 token context · 1 concurrent
Based on source documentation; this setup has not been executed by OpenTools. Memory: unknown. No compatible measurement available.
Read the complete setup guide
Before you start
- Use an owned Ubuntu 24.04 x86_64 Intel CPU host with eight selected threads, 64 GiB RAM and 80 GiB available disk. These are planning allocations; full fit, speed and setup peak remain unknown.
- The selected named GGUF and native metadata declare CC BY-NC 4.0. The actual publisher license incorporates the Cohere Labs Acceptable Use Addendum and policy. NonCommercial means not primarily intended for commercial advantage or monetary compensation; private or paid-VM use is not an exemption. Retain required attribution, license/disclaimer/source notices and indicate modifications when sharing; do not impose additional restrictions on licensed rights. The incorporated policy prohibits specified harmful/illegal/privacy/deceptive uses and synthetic data outputs for commercial purposes, including model or product development. Read all full terms and assess your own intended use before downloading. No terms acceptance, commercial permission or legal clearance is performed for you.
- Use the exact named complete August 2024 R derivative. The frozen R canonical name/API is coherent despite its catalog slug. This is not R+, R7B, a partial adapter or a native FP32 setup.
- Native raw files are gated in the captured access state. Public official metadata includes the exact native template, and the named derivative repository metadata and README are public. A download access error must stop the guide; resolve through the publisher without gate bypass.
- Review this pinned runtime, public source and template before building. Private loopback/key does not grant commercial rights or create a public TLS/multiuser service. Use original non-sensitive non-commercial research text only.
Exact model and requirements
- Artifact:
- Bartowski (named GGUF derivative); Cohere Labs (native model) / Command R (08-2024) Q4_K_M by Bartowski · checkpoint bartowski/c4ai-command-r-08-2024-GGUF
- Publisher revision:
- 0390adf30f4e73160774db7d8fcd1f6172a16e65
- Artifact digest:
- sha256:9fd704afe04c123ce11f9f25709f6b233065853dc346ebc3b7eeff8edc4d20a0
- Download size:
- 18.44 GiB
- Memory requirement (unknown):
- Unknown — No process peak measured for complete Q4_K_M weights, F16 attention cache, scratch, runtime and OS.
- System RAM / disk:
- Unknown / Unknown
- Placement:
- single device. One CPU host with one conversation slot. All model layers run on CPU; no GPU or disk-offload path is configured.
- KV cache:
- F16 K and V attention cache; actual process/cache allocation unmeasured.
- Workload:
- 4,096 token context; Unknown; Unknown; concurrency 1; batch 1; cache unknown; repetitions Unknown.
- Tokenizer:
- Named derivative tokenizer and independently pinned native August R default template · GGUF derivative pin and native template pin; full revisions in download steps
- Sampling / reasoning:
- Temperature 0.3 follows the actual native R August model-card example. Top_k 0, top_p 1, min_p 0 and repeat_penalty 1 are explicit author choices, not a universal publisher recommendation. n_predict 512; native END_OF_TURN stop. No R7B response markers are introduced. Ordinary alternating user/assistant text conversation uses the exact native default template. It emits one BOS and an assistant CHATBOT prefix, with no START_RESPONSE marker. Tool-use/RAG named templates, citations and action loops are distinct workflows and not enabled. Native 128K support is separate from this author-selected 4,096-token walkthrough.
- Runtime prerequisites:
- Ubuntu 24.04 x86_64; See prerequisites above.
The selected named GGUF and native metadata declare CC BY-NC 4.0. The actual publisher license incorporates the Cohere Labs Acceptable Use Addendum and policy. NonCommercial means not primarily intended for commercial advantage or monetary compensation; private or paid-VM use is not an exemption. Retain required attribution, license/disclaimer/source notices and indicate modifications when sharing; do not impose additional restrictions on licensed rights. The incorporated policy prohibits specified harmful/illegal/privacy/deceptive uses and synthetic data outputs for commercial purposes, including model or product development. Read all full terms and assess your own intended use before downloading. No terms acceptance, commercial permission or legal clearance is performed for you.
- The exact complete GGUF file is 19,800,833,472 bytes. Publisher native metadata reports 32,296,476,672 F16 parameters and a 32B release label; this is the August R model, not the older rounded 35B model or R+.
- 64 GiB RAM, 80 GiB disk and eight CPU threads/vCPUs are planning allocations. Artifact size is not process-memory fit. Native FP32 expansion would require a separately qualified larger-host path and is not silently claimed here.
One-time setup and hardware
- hardware
- Unknown — Existing desktop purchase price and incremental hardware cost depend on the reader; unknown.
- Existing desktop purchase price and incremental hardware cost depend on the reader; unknown. Tax: unknown.
- setup
- Unknown — Initial provisioning/build time and labor cost are unknown.
- Initial provisioning/build time and labor cost are unknown. Tax: unknown.
Recurring costs
- electricity
- Unknown — Actual wall power, electricity rate and powered hours are unknown; no TDP-to-wall-power conversion.
- Actual wall power, electricity rate and powered hours are unknown; no TDP-to-wall-power conversion. Tax: unknown.
- storage
- Unknown — Incremental cost of local disk capacity is unknown. The 80 GiB free-disk walkthrough allocation is an author choice, not a measured requirement.
- Incremental cost of local disk capacity is unknown. The 80 GiB free-disk walkthrough allocation is an author choice, not a measured requirement. Tax: unknown.
- egress
- Unknown — Data transfer price, allowances and traffic volume are unknown.
- Data transfer price, allowances and traffic volume are unknown. Tax: unknown.
- administration
- Unknown — Maintenance labor rate and hours are unknown.
- Maintenance labor rate and hours are unknown. Tax: unknown.
Install the runtime
1. Build the exact CPU runtime
Install host build dependencies and compile the pinned official source. No CUDA driver is required. Build and run natively on this same CPU; save resolved package/tool versions and build logs during actual execution.
Commandset -e sudo apt-get update sudo apt-get install -y build-essential cmake git curl ca-certificates openssl python3 mkdir -p "$HOME/command-r0824-hosting/models" cd "$HOME/command-r0824-hosting" git clone https://github.com/ggml-org/llama.cpp.git git -C llama.cpp checkout --detach 620d2225cc436ab0b001e3200e590e3b286ef9ab cmake -S llama.cpp -B llama.cpp/build -DCMAKE_BUILD_TYPE=Release -DGGML_METAL=OFF -DGGML_NATIVE=ON cmake --build llama.cpp/build --config Release -j 4 ./llama.cpp/build/bin/llama-server --versionExpected result: Checkout and build succeed; server version identifies the pinned source. Stop on failure. No inference has been executed by the author.
Download the model
1. Download the exact complete derivative and native template
Read full NC/Addendum/AUP terms first. The weight checksum fails fast before metadata work. Public revision-addressed API identity/revision and exact template bytes are checked; mutable counters are not byte-pinned. The native and derivative revisions are independent pins, not a claim of byte equivalence or known quantizer source commit.
Commandset -e cd "$HOME/command-r0824-hosting" curl --fail --location --retry 0 'https://huggingface.co/bartowski/c4ai-command-r-08-2024-GGUF/resolve/0390adf30f4e73160774db7d8fcd1f6172a16e65/c4ai-command-r-08-2024-Q4_K_M.gguf' -o models/command-r0824-Q4_K_M.gguf echo '9fd704afe04c123ce11f9f25709f6b233065853dc346ebc3b7eeff8edc4d20a0 models/command-r0824-Q4_K_M.gguf' | sha256sum --check test "$(stat -c %s models/command-r0824-Q4_K_M.gguf)" = 19800833472 curl --fail --location 'https://huggingface.co/api/models/CohereLabs/c4ai-command-r-08-2024/revision/dc835b893cd3fb8f14b24970dbc2a0a6d3c22ee3?blobs=true' -o models/native-metadata.json python3 - <<'PYTEMPLATE' import hashlib,json,pathlib d=json.loads(pathlib.Path('models/native-metadata.json').read_text()) assert d['id']=='CohereLabs/c4ai-command-r-08-2024' and d['sha']=='dc835b893cd3fb8f14b24970dbc2a0a6d3c22ee3' t=next(x['template'] for x in d['config']['tokenizer_config']['chat_template'] if x['name']=='default') b=t.encode('utf-8');assert len(b)==1254 and hashlib.sha256(b).hexdigest()=='6269c0952a633ea8ec6ac9e7cc28e7af15295b0819fd7c09fab3f0e6a392c7e1' pathlib.Path('models/command-default.jinja').write_bytes(b) print('Native template length/hash verified') PYTEMPLATEExpected result: Complete weight byte/checksum and exact 1,254-byte native template checks pass. Stop on any mismatch or access error.
Configure
1. Create a private service credential
Generate a local server password on your own host when following this guide. Preserve an existing key on repeat setup and never print or share it. This is not an provider API key.
Commandset -e cd "$HOME/command-r0824-hosting" umask 077 if [ ! -e .api-key ]; then openssl rand -hex 32 > .api-key fi test -s .api-key || { echo 'Restore the empty service-key file before continuing.'; exit 1; } chmod 600 .api-keyExpected result: A nonempty private .api-key exists with mode 600. Existing key preserved; no secret printed.
Start the model
1. Start the private native August R CPU backend
Run in the foreground. One slot and 4,096 total context tokens include the full native prompt and up to 512 output tokens. Select the pinned native default template; no R7B/generic template is substituted. Stop if native architecture or template loading fails.
Commandset -e cd "$HOME/command-r0824-hosting" COMMAND_SERVICE_KEY="$(cat .api-key)" test -n "$COMMAND_SERVICE_KEY" export COMMAND_SERVICE_KEY ./llama.cpp/build/bin/llama-server -m models/command-r0824-Q4_K_M.gguf --alias command-r0824 --host 127.0.0.1 --port 8080 --api-key "$COMMAND_SERVICE_KEY" --device none -ngl 0 -t 8 -c 4096 -np 1 --cache-type-k f16 --cache-type-v f16 --jinja --chat-template-file models/command-default.jinja --skip-chat-parsing --reasoning-format none --temp 0.3 --top-k 0 --top-p 1 --min-p 0 --repeat-penalty 1Expected result: The native COMMAND_R backend loads and listens only on loopback. Privileged users may see process arguments. Actual successful model load, fit and speed remain unmeasured.
Connect to your endpoint
http://127.0.0.1:8080/completion
Trusted desktop loopback only; checks run in a second local terminal.
Access: loopback. Reader-generated private Bearer key protects native template, tokenization and completion routes. Native health is public on loopback. No Cohere provider API credential or paid inference API is used.
Verify the setup
1. Check a complete native non-commercial conversation
Use a second terminal or SSH session on this host. The original library-catalog question and one-sentence follow-up test native conversation formatting and normal completion. Inspect both actual answers; protocol success does not establish correctness or quality. Client timeout does not cancel native generation.
Commandset -e cd "$HOME/command-r0824-hosting" cat > chat-client.py <<'PYCLIENT' """Reader-run native Command R August 2024 ordinary text conversation check.""" import json,pathlib,urllib.request def check_messages(messages): if not isinstance(messages,list) or not messages or len(messages)%2!=1: raise ValueError('user_ending_conversation_required') for i,m in enumerate(messages): if not isinstance(m,dict) or set(m)!={'role','content'} or m['role']!=('user' if i%2==0 else 'assistant') or not isinstance(m['content'],str) or not m['content'].strip() or len(m['content'].encode())>8192 or '<|' in m['content'] or '<BOS_TOKEN>' in m['content']: raise ValueError('invalid_plain_text_conversation') def run_case(messages,post): check_messages(messages) prompt=post('/apply-template',{'messages':messages})['prompt'] if not isinstance(prompt,str) or not prompt.startswith('<BOS_TOKEN>') or prompt.count('<BOS_TOKEN>')!=1 or not prompt.endswith('<|START_OF_TURN_TOKEN|><|CHATBOT_TOKEN|>'): raise ValueError('native_template_mismatch') ids=post('/tokenize',{'content':prompt,'add_special':False,'parse_special':True})['tokens'] if not isinstance(ids,list) or not ids or any(type(i)!=int or i<0 for i in ids) or len(ids)>3584: raise ValueError('prompt_budget_or_token_shape') result=post('/completion',{'prompt':ids,'n_predict':512,'temperature':0.3,'top_k':0,'top_p':1.0,'min_p':0.0,'repeat_penalty':1.0,'stop':['<|END_OF_TURN_TOKEN|>'],'return_tokens':True,'cache_prompt':False}) if result.get('stop_type') not in ('eos','word') or result.get('truncated') is not False: raise ValueError('unfinished_or_truncated_response') tokens=result.get('tokens');text=result.get('content') if not isinstance(tokens,list) or not 1<=len(tokens)<=512 or any(type(i)!=int or i<0 for i in tokens) or not isinstance(text,str) or not text.strip() or '<|' in text: raise ValueError('response_shape') return {'prompt_tokens':len(ids),'generated_tokens':len(tokens),'response':text} def main(): key=pathlib.Path('.api-key').read_text().strip() def post(path,data): req=urllib.request.Request('http://127.0.0.1:8080'+path,json.dumps(data).encode(),headers={'Authorization':'Bearer '+key,'Content-Type':'application/json'}) with urllib.request.urlopen(req,timeout=600) as response:return json.load(response) first=[{'role':'user','content':'Explain in two short sentences why a library catalog helps a reader find a book.'}] answer=run_case(first,post);print(json.dumps(answer,ensure_ascii=False)) follow=first+[{'role':'assistant','content':answer['response']},{'role':'user','content':'Summarize your explanation in one sentence.'}] print(json.dumps(run_case(follow,post),ensure_ascii=False)) print('Inspect both answers for relevance and coherent follow-up. Syntax and stop checks do not establish correctness or quality.') if __name__=='__main__':main() PYCLIENT python3 chat-client.py code=$(curl -sS -o unauthorized.json -w '%{http_code}' -H 'Content-Type: application/json' --data '{"content":"test"}' http://127.0.0.1:8080/tokenize) test "$code" = 401Expected result: Two actual nonempty responses with counted prompts and normal stops; unauthenticated tokenization returns 401. Human records relevance and coherent follow-up without a forced gold answer.
Performance evidence
No compatible measurement available. Documented support and estimated memory do not establish response speed.
Limits and troubleshooting
- Documented source-qualified setup only; no weight body, build, model load, inference, memory fit, CPU throughput or answer quality was measured.
- The selected named GGUF and native metadata declare CC BY-NC 4.0. The actual publisher license incorporates the Cohere Labs Acceptable Use Addendum and policy. NonCommercial means not primarily intended for commercial advantage or monetary compensation; private or paid-VM use is not an exemption. Retain required attribution, license/disclaimer/source notices and indicate modifications when sharing; do not impose additional restrictions on licensed rights. The incorporated policy prohibits specified harmful/illegal/privacy/deceptive uses and synthetic data outputs for commercial purposes, including model or product development. Read all full terms and assess your own intended use before downloading. No terms acceptance, commercial permission or legal clearance is performed for you.
- This frozen R record is Command R (08-2024), API cohere/command-r-08-2024. The separate frozen R+ record is Command R+ (08-2024), API cohere/command-r-plus-08-2024; no merging or slug-based identity change is performed.
- Publisher native metadata reports 32,296,476,672 F16 parameters; the native public model card labels this release 32B and 128K context. Gated raw config dimensions are not inferred from Transformers defaults or older R models.
- The independently pinned native default template is 1,254 bytes and ends at the assistant CHATBOT prefix. It differs materially from R7B; no START_RESPONSE/END_RESPONSE wrapper is copied. Native tool/RAG templates and action/citation workflows remain open.
- The public Bartowski README identifies the exact native base repository and llama.cpp b3634 quantizer, but does not pin its input base commit. Independently pinned native template and derivative must pass the reader functional checks; native weight-byte equivalence and exact quantizer input revision are not claimed.
- 4,096 total context and 512 output cap are author choices. Actual prompt/output counts are observed only at reader execution. Native 128K context does not establish RAM fit at that context.
- Full complete Q4_K_M weights and F16 cache, scratch/loading transients and OS memory all matter. The 64 GiB allocation is not a measured minimum; native FP32 and its larger hardware needs remain separately unqualified.
- Private loopback/auth does not isolate privileged users or supply public TLS, multiuser resilience or an inference deadline. Broader native full-precision, GPU, context, RAG/tool and R+ routes remain open; required guide total is null.
- Access error or license uncertainty
- Stop and resolve the exact publisher/derivative access and full terms. Do not bypass a gate, create a new agent credential or assume private hosting or VM payment grants commercial model rights.
- Native template, BOS or architecture mismatch
- Verify exact runtime, complete derivative digest and native template extraction. Do not substitute R7B response markers, a generic template or extra BOS.
- Prompt overflow or partial response
- Shorten the complete conversation within the 3,584-token prompt budget. Limit stop or truncation fails the functional check. A changed context/output budget requires requalification.
- 401 or connection refused
- Check foreground readiness, matching private key and same-host loopback or correctly established SSH tunnel. Keep public port 8080 closed.
- OOM, slow or poor response
- Stop and record actual process memory/resources and answers. Planning allocation, artifact size or normal stop do not prove fit, speed or quality.
Stop the model
1. Stop the CPU backend
Press Ctrl+C in the foreground server terminal. Unset the private key in each shell and close any SSH tunnel. Verify loopback port 8080 closes.
Commandunset COMMAND_SERVICE_KEYExpected result: The backend stops serving loopback port 8080.
Uninstall
1. Remove the dedicated walkthrough files
Back up wanted outputs, confirm the exact path and remove only this dedicated directory. Shared system packages remain.
Commandrm -r -- "$HOME/command-r0824-hosting"Expected result: Dedicated complete weights, build, native metadata/template, client, private key and outputs removed.
Sources and verification
Sources checked Oct 16, 2026. Verification source
- https://huggingface.co/api/models/CohereForAI/c4ai-command-r-08-2024?blobs=truepublisher · retrieved Oct 16, 2026 · revision dc835b893cd3
- https://docs.cohere.com/docs/command-rpublisher · retrieved Oct 16, 2026
- https://docs.cohere.com/docs/command-r-pluspublisher · retrieved Oct 16, 2026
- https://cohere.com/c4ai-cc-by-nc-licensepublisher · retrieved Oct 16, 2026
- https://creativecommons.org/licenses/by-nc/4.0/legalcode.enpublisher · retrieved Oct 16, 2026
- https://huggingface.co/api/models/bartowski/c4ai-command-r-08-2024-GGUF?blobs=truepublisher · retrieved Oct 16, 2026 · revision 0390adf30f4e
- https://docs.cohere.com/docs/cohere-labs-acceptable-use-policypublisher · retrieved Oct 16, 2026
- https://huggingface.co/bartowski/c4ai-command-r-08-2024-GGUF/raw/0390adf30f4e73160774db7d8fcd1f6172a16e65/README.mdpublisher · retrieved Oct 16, 2026 · revision 0390adf30f4e
- https://raw.githubusercontent.com/ggml-org/llama.cpp/620d2225cc436ab0b001e3200e590e3b286ef9ab/src/models/command-r.cppruntime maintainer · retrieved Oct 16, 2026 · revision 620d2225
- https://raw.githubusercontent.com/ggml-org/llama.cpp/620d2225cc436ab0b001e3200e590e3b286ef9ab/convert_hf_to_gguf.pyruntime maintainer · retrieved Oct 16, 2026 · revision 620d2225
- https://huggingface.co/CohereLabs/c4ai-command-r-08-2024publisher · retrieved Oct 16, 2026 · revision dc835b893cd3
- https://huggingface.co/api/models/CohereLabs/c4ai-command-r-08-2024/revision/dc835b893cd3fb8f14b24970dbc2a0a6d3c22ee3?blobs=truepublisher · retrieved Oct 16, 2026 · revision dc835b893cd3
- https://huggingface.co/api/models/bartowski/c4ai-command-r-08-2024-GGUF/revision/0390adf30f4e73160774db7d8fcd1f6172a16e65?blobs=truepublisher · retrieved Oct 16, 2026 · revision 0390adf30f4e
- https://raw.githubusercontent.com/ggml-org/llama.cpp/620d2225cc436ab0b001e3200e590e3b286ef9ab/conversion/command_r.pyruntime maintainer · retrieved Oct 16, 2026 · revision 620d2225
- https://opentools.ai/llms/command-r-08-2024-cohereopentools · retrieved Oct 16, 2026 · revision august-r-native-client-v1
- https://www.intel.com/content/www/us/en/products/sku/134586/intel-core-i512400-processor-18m-cache-up-to-4-40-ghz/specifications.htmlmanufacturer · retrieved Oct 16, 2026
- https://raw.githubusercontent.com/ggml-org/llama.cpp/620d2225cc436ab0b001e3200e590e3b286ef9ab/docs/build.mdruntime maintainer · retrieved Oct 16, 2026 · revision 620d2225cc436ab0b001e3200e590e3b286ef9ab
- https://raw.githubusercontent.com/ggml-org/llama.cpp/620d2225cc436ab0b001e3200e590e3b286ef9ab/ggml/src/ggml-cpu/CMakeLists.txtruntime maintainer · retrieved Oct 16, 2026 · revision 620d2225cc436ab0b001e3200e590e3b286ef9ab
- https://docs.liquid.ai/lfm/inference/llama-cppruntime maintainer · retrieved Oct 16, 2026
- https://raw.githubusercontent.com/ggml-org/llama.cpp/620d2225cc436ab0b001e3200e590e3b286ef9ab/tools/server/README.mdruntime maintainer · retrieved Oct 16, 2026 · revision 620d2225cc436ab0b001e3200e590e3b286ef9ab
- https://raw.githubusercontent.com/ggml-org/llama.cpp/620d2225cc436ab0b001e3200e590e3b286ef9ab/common/arg.cppruntime maintainer · retrieved Oct 16, 2026 · revision 620d2225cc436ab0b001e3200e590e3b286ef9ab
- https://api.github.com/repos/ggml-org/llama.cpp/releases/latestruntime maintainer · retrieved Oct 16, 2026