Local hardwareDocumented setupSources checked Oct 16, 2026
Run Command R7B ordinary text chat on a 32 GiB Intel desktop
Use the exact Bartowski Q4_K_M derivative of Cohere Command R7B (12-2024), pinned native default template and CPU llama.cpp for a private non-commercial text conversation. Four threads, one sequence, 4,096 total context tokens and a 512-token output cap; native prompt counting, single-BOS checks and complete-response checks make the boundary explicit. Memory fit, speed and answer quality remain unmeasured.
- Hardware
- Intel Core i5-12400 desktop, 4 inference threads, 32 GiB RAM
- Runtime
- llama.cpp v0.7.0 (620d2225cc436ab0b001e3200e590e3b286ef9ab) · CPU
- Model format
- GGUF · Q4_K_M mixed GGUF quantization · Bartowski Q4_K_M GGUF derivative
- Workload
- 4,096 token context · 1 concurrent
Based on source documentation; this setup has not been executed by OpenTools. Memory: unknown. No compatible measurement available.
Read the complete setup guide
Before you start
- Use an owned Intel Core i5-12400 desktop with Ubuntu 24.04 x86_64, 32 GiB installed RAM and 40 GiB free disk. Allocate four inference threads and build natively on the same host.
- Full memory fit, setup disk peak, CPU speed and answer quality for this exact configuration are unknown. Keep the host trusted and do not process sensitive material without a separate security assessment.
- The selected GGUF and native metadata declare CC BY-NC 4.0; the publisher license includes an Acceptable Use Addendum and the model card requires Cohere Labs Acceptable Use Policy. Use only where your intended use satisfies non-commercial terms (not primarily intended for commercial advantage or monetary compensation), attribution/notice, indication-of-changes and no additional restrictions obligations, plus the linked acceptable-use conditions. Paying for an AWS VM does not grant commercial model rights. Read all applicable terms yourself; no terms acceptance, eligibility finding or legal clearance is performed on your behalf.
- Use the December 2024 Command R7B release and exact named GGUF. The native checkpoint is gated; public metadata and its embedded template are available. The selected Bartowski derivative metadata is ungated and its pinned README is public. If its weight download requests access, stop and resolve access through the publisher; do not bypass it.
- This recipe exercises ordinary text conversation only. Publisher text-only model documentation describes 128K context and three 4,096-window sliding-attention layers per global-attention layer; this author-selected 4K walkthrough does not qualify long-context, tool-use or RAG behavior.
Exact model and requirements
- Artifact:
- Bartowski (GGUF derivative); Cohere/CohereForAI (native model) / Command R7B (12-2024) Q4_K_M by Bartowski · checkpoint bartowski/c4ai-command-r7b-12-2024-GGUF
- Publisher revision:
- bfc7a934c45cb839d84c8ca01d87f1cfa51aaa3f
- Artifact digest:
- sha256:0653c1915f7e4819a980deea8e9ba86b0a78cbb283712785cbf091c3ffbe3b13
- Download size:
- 4.71 GiB
- Memory requirement (unknown):
- Unknown — Exact Q4_K_M weights plus F16 sliding/global cache, scratch, runtime and OS memory have not been measured.
- System RAM / disk:
- Unknown / Unknown
- Placement:
- single device. One CPU host; no tensor or pipeline sharding. CPU-only language model; --device none and zero GPU layers.
- KV cache:
- F16 attention K and V; full sliding/global cache and process memory unmeasured.
- Workload:
- 4,096 token context; Unknown; Unknown; concurrency 1; batch 1; cache unknown; repetitions Unknown.
- Tokenizer:
- Exact derivative tokenizer and independently pinned official native default template · bfc7a934c45cb839d84c8ca01d87f1cfa51aaa3f; native template 4f3d0aa6856e322f2f4480fe65420d5d53d297b8
- Sampling / reasoning:
- Temperature 0.3 follows the native model-card example. Top_k 0, top_p 1, min_p 0, repeat_penalty 1 are author choices to make other filters explicit, not a publisher universal recommendation. n_predict 512; stop END_RESPONSE or END_OF_TURN. Ordinary native text conversation with no tools/documents. The exact default template selects its plain conversation branch and emits START_RESPONSE. Native tool/RAG templates and thinking/action parsing are distinct workflows, not enabled by this recipe.
- Runtime prerequisites:
- Ubuntu 24.04 x86_64; See prerequisites above.
The selected GGUF and native metadata declare CC BY-NC 4.0; the publisher license includes an Acceptable Use Addendum and the model card requires Cohere Labs Acceptable Use Policy. Use only where your intended use satisfies non-commercial terms (not primarily intended for commercial advantage or monetary compensation), attribution/notice, indication-of-changes and no additional restrictions obligations, plus the linked acceptable-use conditions. Paying for an AWS VM does not grant commercial model rights. Read all applicable terms yourself; no terms acceptance, eligibility finding or legal clearance is performed on your behalf.
- The derivative file is 5,057,009,792 bytes. Native API reports 8,028,033,024 BF16 parameters despite the release name R7B; no active-expert shortcut is used.
- 32 GiB RAM, 40 GiB disk and four threads are planning allocations, not measured minima or a fit guarantee. Native 128K context does not establish RAM fit at that context.
One-time setup and hardware
- hardware
- Unknown — Existing desktop purchase price and incremental hardware cost depend on the reader; unknown.
- Existing desktop purchase price and incremental hardware cost depend on the reader; unknown. Tax: unknown.
- setup
- Unknown — Initial provisioning/build time and labor cost are unknown.
- Initial provisioning/build time and labor cost are unknown. Tax: unknown.
Recurring costs
- electricity
- Unknown — Actual wall power, electricity rate and powered hours are unknown; no TDP-to-wall-power conversion.
- Actual wall power, electricity rate and powered hours are unknown; no TDP-to-wall-power conversion. Tax: unknown.
- storage
- Unknown — Incremental cost of local disk capacity is unknown. The 40 GiB free-disk walkthrough allocation is an author choice, not a measured requirement.
- Incremental cost of local disk capacity is unknown. The 40 GiB free-disk walkthrough allocation is an author choice, not a measured requirement. Tax: unknown.
- egress
- Unknown — Data transfer price, allowances and traffic volume are unknown.
- Data transfer price, allowances and traffic volume are unknown. Tax: unknown.
- administration
- Unknown — Maintenance labor rate and hours are unknown.
- Maintenance labor rate and hours are unknown. Tax: unknown.
Install the runtime
1. Build the exact CPU runtime
Install host build dependencies and compile the pinned official source. No CUDA driver is required. Build and run natively on this same CPU; save resolved package/tool versions and build logs during actual execution.
Commandsudo apt-get update sudo apt-get install -y build-essential cmake git curl ca-certificates openssl python3 mkdir -p "$HOME/command-r7b-hosting/models" cd "$HOME/command-r7b-hosting" git clone https://github.com/ggml-org/llama.cpp.git git -C llama.cpp checkout --detach 620d2225cc436ab0b001e3200e590e3b286ef9ab cmake -S llama.cpp -B llama.cpp/build -DCMAKE_BUILD_TYPE=Release -DGGML_METAL=OFF -DGGML_NATIVE=ON cmake --build llama.cpp/build --config Release -j 4 ./llama.cpp/build/bin/llama-server --versionExpected result: Checkout and build succeed; server version identifies the pinned source. Stop on failure. No inference has been executed by the author.
Download the model
1. Download the exact named derivative and native template
Read the linked license, its Acceptable Use Addendum and the publisher acceptable-use policy before downloading. These commands do not accept a gate. The template is extracted verbatim from the official revision-addressed public metadata, whose canonical identity, pinned revision and exact template length/hash are checked; mutable API counters are not byte-pinned. No weight conversion or template rewriting is performed.
Commandset -e cd "$HOME/command-r7b-hosting" curl --fail --location --retry 2 'https://huggingface.co/bartowski/c4ai-command-r7b-12-2024-GGUF/resolve/bfc7a934c45cb839d84c8ca01d87f1cfa51aaa3f/c4ai-command-r7b-12-2024-Q4_K_M.gguf' -o models/command-r7b-Q4_K_M.gguf echo '0653c1915f7e4819a980deea8e9ba86b0a78cbb283712785cbf091c3ffbe3b13 models/command-r7b-Q4_K_M.gguf' | sha256sum --check curl --fail --location 'https://huggingface.co/api/models/CohereForAI/c4ai-command-r7b-12-2024/revision/4f3d0aa6856e322f2f4480fe65420d5d53d297b8?blobs=true' -o models/native-metadata.json python3 - <<'PYTEMPLATE' import hashlib,json,pathlib p=pathlib.Path('models/native-metadata.json') d=json.loads(p.read_text()); assert d['id']=='CohereLabs/c4ai-command-r7b-12-2024' assert d['sha']=='4f3d0aa6856e322f2f4480fe65420d5d53d297b8' t=next(x['template'] for x in d['config']['tokenizer_config']['chat_template'] if x['name']=='default') b=t.encode('utf-8'); assert len(b)==12784 assert hashlib.sha256(b).hexdigest()=='01dfd1eb4bbe2b40f75f522bf91b43c8184fe22de2f071d4bcb80945fed19e60' pathlib.Path('models/command-default.jinja').write_bytes(b) print('Verified exact native default template:',len(b),'bytes') PYTEMPLATEExpected result: Weight checksum and pinned native template length/hash checks pass; template is 12,784 bytes. Stop on any mismatch or access error.
Configure
1. Create a private service credential
Generate a local server password on your own host when following this guide. Preserve an existing key on repeat setup and never print or share it. This is not an provider API key.
Commandcd "$HOME/command-r7b-hosting" umask 077 if [ ! -e .api-key ]; then openssl rand -hex 32 > .api-key fi test -s .api-key || { echo 'Restore the empty service-key file before continuing.'; exit 1; } chmod 600 .api-keyExpected result: A nonempty private .api-key exists with mode 600. Existing key preserved; no secret printed.
Start the model
1. Start the private native CPU conversation backend
Run in the foreground. One slot and 4,096 context tokens include all native prompt tokens and up to 512 output tokens. The exact native template is explicitly selected; no generic template or vendor API is substituted. Stop if native architecture or template loading fails.
Commandcd "$HOME/command-r7b-hosting" export COMMAND_SERVICE_KEY="$(cat .api-key)" ./llama.cpp/build/bin/llama-server -m models/command-r7b-Q4_K_M.gguf --alias command-r7b --host 127.0.0.1 --port 8080 --api-key "$COMMAND_SERVICE_KEY" --device none -ngl 0 -t 4 -c 4096 -np 1 --cache-type-k f16 --cache-type-v f16 --jinja --chat-template-file models/command-default.jinja --skip-chat-parsing --reasoning-format none --temp 0.3 --top-k 0 --top-p 1 --min-p 0 --repeat-penalty 1Expected result: The pinned Cohere2 backend loads and listens on loopback. Privileged users may see process arguments. No successful model load or inference is preclaimed.
Connect to your endpoint
http://127.0.0.1:8080/completion
Same trusted desktop loopback only.
Access: loopback. Reader-generated private Bearer key protects template, tokenization and completion routes; health is public on loopback. No paid provider API credential is used.
Verify the setup
1. Verify full native prompt accounting and two conversation turns
Use a second terminal on the same host (on AWS, a second SSH session to the VM). The client calls the native template route, checks one BOS and START_RESPONSE, tokenizes with add_special false and parse_special true, then sends those integer IDs to raw completion to avoid another BOS. It reserves 512 output tokens and rejects length stops, truncation or leaked control markers. Inspect both actual answers for relevance and coherent follow-up; the protocol check does not prove answer quality.
Commandcd "$HOME/command-r7b-hosting" curl --fail http://127.0.0.1:8080/health cat > chat_check.py <<'PYCLIENT' """Reader-run native Command R7B ordinary text conversation check.""" import json,pathlib,urllib.request def check_messages(messages): if not isinstance(messages,list) or not messages or len(messages)%2!=1: raise ValueError('user_ending_conversation_required') for i,m in enumerate(messages): if not isinstance(m,dict) or set(m)!={'role','content'} or m['role']!=('user' if i%2==0 else 'assistant') or not isinstance(m['content'],str) or not m['content'].strip() or len(m['content'].encode())>8192 or '<|' in m['content'] or '<BOS_TOKEN>' in m['content']: raise ValueError('invalid_plain_text_conversation') def run_case(messages,post): check_messages(messages) prompt=post('/apply-template',{'messages':messages})['prompt'] if not isinstance(prompt,str) or not prompt.startswith('<BOS_TOKEN>') or prompt.count('<BOS_TOKEN>')!=1 or not prompt.endswith('<|START_OF_TURN_TOKEN|><|CHATBOT_TOKEN|><|START_RESPONSE|>'): raise ValueError('native_template_mismatch') ids=post('/tokenize',{'content':prompt,'add_special':False,'parse_special':True})['tokens'] if not isinstance(ids,list) or not ids or any(type(i)!=int or i<0 for i in ids) or len(ids)>3584: raise ValueError('prompt_budget_or_token_shape') result=post('/completion',{'prompt':ids,'n_predict':512,'temperature':0.3,'top_k':0,'top_p':1.0,'min_p':0.0,'repeat_penalty':1.0,'stop':['<|END_RESPONSE|>','<|END_OF_TURN_TOKEN|>'],'return_tokens':True,'cache_prompt':False}) if result.get('stop_type') not in ('eos','word') or result.get('truncated') is not False: raise ValueError('unfinished_or_truncated_response') tokens=result.get('tokens');text=result.get('content') if not isinstance(tokens,list) or not 1<=len(tokens)<=512 or any(type(i)!=int or i<0 for i in tokens) or not isinstance(text,str) or not text.strip() or '<|' in text: raise ValueError('response_shape') return {'prompt_tokens':len(ids),'generated_tokens':len(tokens),'response':text} def main(): key=pathlib.Path('.api-key').read_text().strip() def post(path,data): req=urllib.request.Request('http://127.0.0.1:8080'+path,json.dumps(data).encode(),headers={'Authorization':'Bearer '+key,'Content-Type':'application/json'}) with urllib.request.urlopen(req,timeout=600) as response:return json.load(response) first=[{'role':'user','content':'Explain in two short sentences why a library catalog helps a reader find a book.'}] answer=run_case(first,post);print(json.dumps(answer,ensure_ascii=False)) follow=first+[{'role':'assistant','content':answer['response']},{'role':'user','content':'Summarize your explanation in one sentence.'}] print(json.dumps(run_case(follow,post),ensure_ascii=False)) print('Inspect both answers for relevance and coherent follow-up. Syntax and stop checks do not establish correctness or quality.') if __name__=='__main__':main() PYCLIENT python3 chat_check.py curl -s -o /dev/null -w '%{http_code}\n' http://127.0.0.1:8080/completion -H 'Content-Type: application/json' -d '{"prompt":"test","n_predict":1}' unset COMMAND_SERVICE_KEYExpected result: Ready health; two nonempty response records with actual prompt/generated counts and normal stop. A complete second turn preserves first-turn context. Missing key returns 401. No exact answer or measured quality is preclaimed.
Performance evidence
No compatible measurement available. Documented support and estimated memory do not establish response speed.
Limits and troubleshooting
- Documented source-backed setup only; no weights downloaded, build, model load, inference, memory fit, throughput or quality measurements performed.
- The selected GGUF and native metadata declare CC BY-NC 4.0; the publisher license includes an Acceptable Use Addendum and the model card requires Cohere Labs Acceptable Use Policy. Use only where your intended use satisfies non-commercial terms (not primarily intended for commercial advantage or monetary compensation), attribution/notice, indication-of-changes and no additional restrictions obligations, plus the linked acceptable-use conditions. Paying for an AWS VM does not grant commercial model rights. Read all applicable terms yourself; no terms acceptance, eligibility finding or legal clearance is performed on your behalf.
- Native model metadata names Cohere2ForCausalLM and reports 8,028,033,024 BF16 parameters. The published R7B name and model-card rounded 7B description are preserved without silently changing the release identity. This is text input/output.
- Direct native README/config/tokenizer/generation downloads return 401. The public rendered native model card and official metadata with exact embedded templates remain available. Raw config dimensions have not been independently read; Transformers defaults are not asserted as checkpoint values.
- The exact native default template is extracted from public revision-addressed API metadata. The tools/documents branches, tool/RAG named templates, citations and action loops require separate qualification.
- The full prompt is counted after native rendering. 4,096 total tokens is an author-selected reduced context, not a new publisher maximum; 512 is an output cap. Native 128K support does not establish this CPU host fits 128K.
- F16 cache is explicit, but three sliding layers per global-attention layer, runtime cache allocation, context and buffers all affect memory. Artifact size alone does not prove RAM fit.
- The public derivative declares the exact native base repository, but does not pin its source revision. The native template and derivative are independently pinned here; functional compatibility must pass the reader checks, not assumed byte-equivalent to native weights.
- Temperature 0.3 follows the native model-card example; other filter settings are explicit author choices. Model responses remain stochastic and may be inaccurate. Normal stop and coherent syntax are not correctness benchmarks.
- 32 GiB RAM, 40 GiB disk and four threads are planning allocations. Original hardware/offer source dates and unknown disk peak, egress, tax, local power and administration costs remain distinct.
- Private loopback/key does not isolate privileged users or make a public/TLS/multiuser service. Broader useful CPU/GPU, context, RAG/tool and production qualification remains open.
- Download access error or license uncertainty
- Stop. Read the exact native and derivative terms and resolve access through the publisher. Do not bypass a gate or assume the paid VM grants commercial model rights.
- Template/BOS or architecture check fails
- Check the exact pinned runtime, derivative hash and native metadata/template extraction. Do not substitute a generic chat template or add another BOS.
- Input too long or response unfinished
- Shorten the conversation so the complete native prompt remains at most 3,584 tokens. A limit stop or truncation fails; changing context/output is a different configuration requiring validation.
- HTTP 401 or connection refused
- Check foreground readiness, matching private key, same-host loopback and the optional SSH tunnel. Keep EC2 public port 8080 closed.
- OOM or slow generation
- Stop and inspect exact process memory and host resources. Allocations and artifact size do not prove fit; no CPU throughput is promised.
Stop the model
1. Stop the CPU backend
Press Ctrl+C in the foreground server terminal. Unset the key in each shell and close any SSH tunnel. Check that loopback port 8080 closes.
Commandunset COMMAND_SERVICE_KEYExpected result: The backend stops serving loopback port 8080.
Uninstall
1. Remove the dedicated walkthrough files
Back up wanted outputs, confirm the exact path and remove only this dedicated directory. Shared system packages remain.
Commandrm -r -- "$HOME/command-r7b-hosting"Expected result: Dedicated weights, build, metadata/template, client and private key files are removed.
Sources and verification
Sources checked Oct 16, 2026. Verification source
- https://huggingface.co/api/models/CohereForAI/c4ai-command-r7b-12-2024?blobs=truepublisher · retrieved Oct 16, 2026 · revision 4f3d0aa6856e322f2f4480fe65420d5d53d297b8
- https://docs.cohere.com/docs/command-r7bpublisher · retrieved Oct 16, 2026
- https://raw.githubusercontent.com/ggml-org/llama.cpp/620d2225/src/models/cohere2.cppruntime maintainer · retrieved Oct 16, 2026 · revision 620d2225cc436ab0b001e3200e590e3b286ef9ab
- https://raw.githubusercontent.com/ggml-org/llama.cpp/620d2225/convert_hf_to_gguf.pyruntime maintainer · retrieved Oct 16, 2026 · revision 620d2225cc436ab0b001e3200e590e3b286ef9ab
- https://cohere.com/c4ai-cc-by-nc-licensepublisher · retrieved Oct 16, 2026
- https://huggingface.co/CohereForAI/c4ai-command-r7b-12-2024publisher · retrieved Oct 16, 2026
- https://raw.githubusercontent.com/huggingface/transformers/v4.57.6/src/transformers/models/cohere2/configuration_cohere2.pyruntime maintainer · retrieved Oct 16, 2026
- https://raw.githubusercontent.com/huggingface/transformers/v4.57.6/src/transformers/models/cohere2/modeling_cohere2.pyruntime maintainer · retrieved Oct 16, 2026
- https://creativecommons.org/licenses/by-nc/4.0/legalcode.enpublisher · retrieved Oct 16, 2026
- https://huggingface.co/api/models/bartowski/c4ai-command-r7b-12-2024-GGUF?blobs=truepublisher · retrieved Oct 16, 2026 · revision bfc7a934c45cb839d84c8ca01d87f1cfa51aaa3f
- https://api.github.com/repos/ggml-org/llama.cpp/contents/conversion?ref=620d2225runtime maintainer · retrieved Oct 16, 2026 · revision 620d2225cc436ab0b001e3200e590e3b286ef9ab
- https://docs.cohere.com/docs/usage-policypublisher · retrieved Oct 16, 2026
- https://huggingface.co/bartowski/c4ai-command-r7b-12-2024-GGUF/raw/bfc7a934c45cb839d84c8ca01d87f1cfa51aaa3f/README.mdpublisher · retrieved Oct 16, 2026 · revision bfc7a934c45cb839d84c8ca01d87f1cfa51aaa3f
- https://raw.githubusercontent.com/ggml-org/llama.cpp/620d2225/conversion/command_r.pyruntime maintainer · retrieved Oct 16, 2026 · revision 620d2225cc436ab0b001e3200e590e3b286ef9ab
- https://docs.cohere.com/docs/cohere-labs-acceptable-use-policypublisher · retrieved Oct 16, 2026
- https://huggingface.co/api/models/CohereForAI/c4ai-command-r7b-12-2024/revision/4f3d0aa6856e322f2f4480fe65420d5d53d297b8?blobs=truepublisher · retrieved Oct 16, 2026 · revision 4f3d0aa6856e322f2f4480fe65420d5d53d297b8
- https://huggingface.co/api/models/bartowski/c4ai-command-r7b-12-2024-GGUF/revision/bfc7a934c45cb839d84c8ca01d87f1cfa51aaa3f?blobs=truepublisher · retrieved Oct 16, 2026 · revision bfc7a934c45cb839d84c8ca01d87f1cfa51aaa3f
- https://opentools.ai/llms/command-r7b-12-2024opentools · retrieved Oct 16, 2026 · revision native-text-client-v1
- https://www.intel.com/content/www/us/en/products/sku/134586/intel-core-i512400-processor-18m-cache-up-to-4-40-ghz/specifications.htmlmanufacturer · retrieved Oct 16, 2026
- https://raw.githubusercontent.com/ggml-org/llama.cpp/620d2225cc436ab0b001e3200e590e3b286ef9ab/ggml/src/ggml-cpu/CMakeLists.txtruntime maintainer · retrieved Oct 16, 2026 · revision 620d2225cc436ab0b001e3200e590e3b286ef9ab
- https://docs.liquid.ai/lfm/inference/llama-cppruntime maintainer · retrieved Oct 16, 2026
- https://raw.githubusercontent.com/ggml-org/llama.cpp/620d2225cc436ab0b001e3200e590e3b286ef9ab/docs/build.mdruntime maintainer · retrieved Oct 16, 2026 · revision 620d2225cc436ab0b001e3200e590e3b286ef9ab
- https://raw.githubusercontent.com/ggml-org/llama.cpp/620d2225cc436ab0b001e3200e590e3b286ef9ab/tools/server/README.mdruntime maintainer · retrieved Oct 16, 2026 · revision 620d2225cc436ab0b001e3200e590e3b286ef9ab
- https://raw.githubusercontent.com/ggml-org/llama.cpp/620d2225cc436ab0b001e3200e590e3b286ef9ab/common/arg.cppruntime maintainer · retrieved Oct 16, 2026 · revision 620d2225cc436ab0b001e3200e590e3b286ef9ab
- https://api.github.com/repos/ggml-org/llama.cpp/releases/latestruntime maintainer · retrieved Oct 16, 2026