Microsoft logo

Phi 4

Phiv4Current
byMicrosoftMicrosoft(enterprise)
Released January 10, 2025
Run this model yourselfAPI DocsPlayground
Context16.4K tokens
Price (In / Out)$0.07/M / $0.14/M
CategoryLarge Language Model
Max Output16.4K tokens

About Phi 4

Phi 4 is a large language model from Microsoft. Supports up to 16,384 token context window. Available from $0.07/M input tokens.

Capabilities

textcode

Input Modalities

text

Output Modalities

text

Technical Details

API Identifier
microsoft/phi-4
Category
Large Language Model
Context Window
16,384 tokens
Max Output Tokens
16,384 tokens

Tags

budget-friendly

Run this model yourself

Local and cloud setup for Phi 4, with exact configurations, sources and clear evidence limits.

Find a model for your hardware
Local hardwareDocumented setupSources checked Oct 15, 2026

Run Phi-4 locally on a Linux L40S server

Serve Microsoft’s original BF16 Phi-4 on one owned NVIDIA L40S with vLLM 0.31.0, a 2,048-token context and one active sequence. The guide pins model files, keeps the endpoint on loopback, and includes verification and removal steps. Documented setup; not executed or benchmarked by OpenTools.

Hardware
One NVIDIA L40S in a Linux x86_64 server
Runtime
vLLM 0.31.0+cu129 · CUDA
Model format
safetensors · BF16 · None; original BF16 weights
Workload
2,048 token context · 1 concurrent

Based on source documentation; this setup has not been executed by OpenTools. Memory: unknown. No compatible measurement available.

Read the complete setup guide

Before you start

  • An already provisioned Linux x86_64 host with one NVIDIA L40S (manufacturer-labeled 48 GB dedicated memory); no claim that this is a tested minimum.
  • Python 3.12 with virtual-environment support, curl, sha256sum, outbound HTTPS to the official package/model hosts, and a working CUDA-compatible NVIDIA driver.
  • The full repository download is about 29.33 GB. Allow additional disk for the virtual environment, package cache and runtime files; complete disk and RAM minima have not been measured.
  • A trusted single-user deployment: this guide binds vLLM to 127.0.0.1. Do not publish port 8000 or treat the API key as protection for every vLLM route.

Exact model and requirements

Artifact:
Microsoft / Phi-4 · checkpoint microsoft/phi-4
Publisher revision:
2db69c1c3e91a05d2c64a3185acfbaf36f744e25
Artifact digest:
The publisher exposes per-file hashes, not one digest for the complete multi-file artifact. Exact weight SHA256 and auxiliary Git blob hashes are listed below.
Download size:
27.31 GiB
Memory requirement (unknown):
Unknown — Complete GPU footprint has not been measured: model weights, KV cache, runtime allocations, compilation and workspace must all fit.
System RAM / disk:
Unknown / Unknown
Placement:
single device. One GPU; no tensor or pipeline sharding and no pooling of GPU and host RAM. No CPU or disk weight offload is enabled.
KV cache:
BF16 via --kv-cache-dtype auto with BF16 model dtype
Workload:
2,048 token context; Unknown; Unknown; concurrency 1; batch 1; cache unknown; repetitions Unknown.
Tokenizer:
microsoft/phi-4 · 2db69c1c3e91a05d2c64a3185acfbaf36f744e25
Sampling / reasoning:
Verification uses temperature 0 and max_tokens 64. The service uses --generation-config vllm; this is not a throughput protocol. Original Phi-4 checkpoint, not a Phi-4 reasoning variant; no special reasoning parser or budget is configured.
Runtime prerequisites:
Linux x86_64; Python 3.12; Working NVIDIA driver R580 or newer; CUDA 12.x applications remain supported by newer drivers through backward compatibility. Verify the selected environment with nvidia-smi and torch.cuda.is_available()..

Official artifact · License

The publisher declares MIT. Read the pinned LICENSE and retain its copyright and permission notice when redistributing covered material.

  • The six weight files total 29,319,042,992 bytes; that is a download measurement from publisher metadata, not the runtime GPU-memory requirement.
  • The selected L40S is labeled 48 GB by the vendor. Dedicated GPU memory and system RAM are different pools and are not summed.
  • No measured peak memory, completed-run evidence, tokens per second or latency result is supplied. A documented runtime architecture match does not prove this exact deployment fits.

One-time setup and hardware

Hardware purchase
Unknown — No dated hardware purchase quote is available; existing ownership is not treated as a zero purchase price.
No dated hardware purchase quote is available; existing ownership is not treated as a zero purchase price. Tax: unknown.
Initial setup work
Unknown — Initial setup time is not priced; no labour rate or duration has been measured.
Initial setup time is not priced; no labour rate or duration has been measured. Tax: unknown.

Recurring costs

Electricity
Unknown — Whole-system measured wall power, powered-on hours and the reader’s electricity tariff are unknown. GPU maximum board power is not a measured system draw.
Whole-system measured wall power, powered-on hours and the reader’s electricity tariff are unknown. GPU maximum board power is not a measured system draw. Tax: unknown.
Operations and maintenance
Unknown — Ongoing patching and operation effort is not measured or priced.
Ongoing patching and operation effort is not measured or priced. Tax: unknown.
Persistent storage
Unknown — The owned host storage arrangement and any recurring backup or storage bill are unknown; no monthly amount is asserted.
The owned host storage arrangement and any recurring backup or storage bill are unknown; no monthly amount is asserted. Tax: unknown.
Data transfer
Unknown — The reader’s internet or network plan and traffic volume are unknown; loopback requests alone do not establish total network costs.
The reader’s internet or network plan and traffic volume are unknown; loopback requests alone do not establish total network costs. Tax: unknown.

Install the runtime

  1. 1. Check the existing GPU host

    Use an already provisioned Linux x86_64 server with one L40S, Python 3.12 with venv support, curl, sha256sum, and a working NVIDIA R580-or-newer driver. Driver provisioning is a server-administrator prerequisite; this guide does not replace the host driver. Keep the machine on a trusted network and ensure port 8000 is unused.

    Command
    nvidia-smi
python3.12 --version
df -h "$HOME"

    Expected result: The L40S is visible, Python reports 3.12, and there is enough free space for the 29.33 GB model download plus the Python environment, package cache and runtime workspace.

  2. 2. Create an isolated environment and install pinned vLLM

    Run in a new shell on the GPU host. The exact CUDA 12.9 vLLM wheel is pinned by release and publisher-provided SHA256. Dependencies resolve inside this fresh environment; do not mix the AMI or host Python environment into it. The system CUDA toolkit version need not equal the bundled wheel runtime. If python3.12 -m venv is unavailable, install your OS vendor’s Python 3.12 venv package first.

    Command
    mkdir -p "$HOME/phi4-hosting"
cd "$HOME/phi4-hosting"
python3.12 -m venv .venv
source .venv/bin/activate
python -m pip install --upgrade pip
python -m pip install --extra-index-url https://download.pytorch.org/whl/cu129 'https://github.com/vllm-project/vllm/releases/download/v0.31.0/vllm-0.31.0%2Bcu129-cp38-abi3-manylinux_2_28_x86_64.whl#sha256=5176ab746f4e736c4a354a11d0620abf0a941de287cad937ac34039293ca6f5f'
python -m pip check
vllm --version
python -c 'import torch; print("CUDA available:", torch.cuda.is_available()); print("GPU:", torch.cuda.get_device_name(0) if torch.cuda.is_available() else "not detected")'

    Expected result: pip check reports no broken requirements; vllm reports version 0.31.0 (the wheel may include +cu129); CUDA available is True and the GPU is NVIDIA L40S. This is an expected check for the reader, not a result observed by OpenTools.

Download the model

  1. 1. Download the exact Microsoft snapshot

    Run after installing the runtime so its Hugging Face dependency is available. The full commit pins weights, configuration and tokenizer together; this does not follow the moving main branch. The official repository was ungated when checked. No model-provider API subscription is needed for this download.

    Command
    cd "$HOME/phi4-hosting"
source .venv/bin/activate
python - <<'PY'
from huggingface_hub import snapshot_download
snapshot_download(repo_id="microsoft/phi-4", revision="2db69c1c3e91a05d2c64a3185acfbaf36f744e25", local_dir="./model")
PY

    Expected result: The command completes and ./model contains config.json, tokenizer files, the safetensors index and all six weight shards from the pinned revision.

  2. 2. Verify every weight shard

    The SHA256 values come from the official repository’s file metadata. Stop if any shard is missing or mismatched; do not proceed with a mixed or partial download.

    Command
    cd "$HOME/phi4-hosting"
sha256sum --check <<'SHA256'
cafbe72210c39b19270cbfce27996afc4383c8a487087bcb91a6290d5ceb8f18  model/model-00001-of-00006.safetensors
9f87a5897162ed988cfb0fac13879634b4bab3da5a756b4f9dd2437ef8145e20  model/model-00002-of-00006.safetensors
19537df78a263aa0c0b2dc12dbce5908021e30a25e528699b08642c3999eeb05  model/model-00003-of-00006.safetensors
d4f17100a39c19d352f2e8108eef6fb975afb86808e50d78a075e162f1edfc93  model/model-00004-of-00006.safetensors
a72666b6001d76e7a9dc3ebe0d05a2c3316f546a8c6779c87343f296bdf82e50  model/model-00005-of-00006.safetensors
a9bfeeea354e31503d09a864e6cd5bd0f1230664065b6a2988b05e22b3228d77  model/model-00006-of-00006.safetensors
SHA256

    Expected result: All six listed safetensors files report OK.

Configure

  1. 1. Create a private local service key

    Run on the GPU host. This writes a fresh random key only when the file is absent, keeps it mode 600 and exports it without printing it. Treat the key file as private. vLLM API-key checks do not cover every route, so keep the loopback binding used below.

    Command
    cd "$HOME/phi4-hosting"
source .venv/bin/activate
umask 077
python - <<'PY'
import secrets
from pathlib import Path
key = Path(".vllm-api-key")
if not key.exists():
    key.write_text(secrets.token_urlsafe(32))
key.chmod(0o600)
PY
export VLLM_API_KEY="$(cat .vllm-api-key)"

    Expected result: A private .vllm-api-key file exists in the dedicated directory; no secret is printed. The shell environment contains VLLM_API_KEY.

Start the model

  1. 1. Start the private Phi-4 endpoint

    Keep this foreground process in its own terminal. ./model is the verified pinned snapshot, so both weights and tokenizer are loaded from it. BF16 weights and automatic BF16 KV cache are used; there is no quantization or weight offload. One sequence is scheduled at a time. Startup can require compilation and workspace; do not infer success until the verification requests pass.

    Command
    cd "$HOME/phi4-hosting"
source .venv/bin/activate
export VLLM_API_KEY="$(cat .vllm-api-key)"
vllm serve ./model \
  --served-model-name phi4-local \
  --host 127.0.0.1 --port 8000 \
  --dtype bfloat16 --kv-cache-dtype auto \
  --max-model-len 2048 --max-num-seqs 1 \
  --tensor-parallel-size 1 --pipeline-parallel-size 1 \
  --generation-config vllm

    Expected result: The process remains running and announces an HTTP listener on 127.0.0.1:8000 after model initialization. If it exits, use the troubleshooting checks before making any success or performance claim.

Connect to your endpoint

http://127.0.0.1:8000/v1

Run clients and verification on the GPU host. Only that host loopback is bound. Remote clients need a separately controlled SSH tunnel; do not change --host to a public interface for convenience.

Access: loopback. VLLM_API_KEY is a reader-generated local service key for /v1, /v2 and /inference paths. Other paths may remain unauthenticated; the loopback boundary is essential. Other users on the same host can reach loopback, so use a trusted single-user machine.

Verify the setup

  1. 1. Check the model identity

    Open a second terminal on the same GPU host. Load the local key without displaying it and call the authenticated models endpoint.

    Command
    cd "$HOME/phi4-hosting"
export VLLM_API_KEY="$(cat .vllm-api-key)"
curl --fail --silent --show-error http://127.0.0.1:8000/v1/models \
  -H "Authorization: Bearer $VLLM_API_KEY"

    Expected result: The JSON response contains the served model id phi4-local. The endpoint responds successfully only after startup completes.

  2. 2. Request a short response

    In that second terminal, send a small chat request. This checks response generation; it is not a benchmark and provides no throughput or latency guarantee.

    Command
    curl --fail --silent --show-error http://127.0.0.1:8000/v1/chat/completions \
  -H "Authorization: Bearer $VLLM_API_KEY" \
  -H 'Content-Type: application/json' \
  -d '{"model":"phi4-local","messages":[{"role":"user","content":"Explain what a GPU does in one sentence."}],"max_tokens":64,"temperature":0}'

    Expected result: A successful JSON response includes a nonempty choices[0].message.content. Wording varies; no exact answer or speed is guaranteed.

  3. 3. Verify the access boundary

    Check that the listener is only on 127.0.0.1:8000 and that the /v1 route rejects a missing key. For AWS, keep the security group closed to inbound 8000 and use the SSH tunnel for remote access.

    Command
    ss -ltn | grep '127.0.0.1:8000'
curl --silent --output /dev/null --write-out '%{http_code}\n' http://127.0.0.1:8000/v1/models

    Expected result: The listener is on loopback and the unauthenticated /v1/models request is denied (normally HTTP 401). This denial does not prove that every server path is authenticated.

Performance evidence

No compatible measurement available. Documented support and estimated memory do not establish response speed.

Limits and troubleshooting

  • This is a documented setup assembled from official model, hardware, runtime and cloud documentation. OpenTools has not executed these commands or verified runtime fit on the selected hardware.
  • The model supports a maximum 16,384-token context in its configuration; this recipe deliberately sets a smaller combined prompt-and-output context. Raising context or concurrency changes the configuration and memory demand.
  • Original Phi-4 can produce incorrect or unsafe output. It is not the Phi-4 mini or reasoning checkpoint. Apply the model publisher’s intended-use and limitation guidance.
  • Dependency packages installed with the pinned vLLM wheel are resolved at install time; capture pip freeze with your run report when reproducibility beyond the pinned runtime wheel is required.
  • Loopback is accessible to other users on the same machine. Use a trusted host. A publicly reachable production service needs a separate authenticated gateway and access design outside this single-user recipe.
nvidia-smi fails or PyTorch reports CUDA unavailable
Check the physical/virtual GPU and host driver before downloading weights. For the AWS path verify the exact supported G6e AMI. Use a fresh environment and the pinned CUDA 12.9 wheel; do not mix a pre-existing torch installation into it.
Installation or download runs out of disk space
Check the filesystem containing $HOME/phi4-hosting and package caches. The model download alone is about 29.33 GB; add space for the environment and runtime files. Resume the same pinned snapshot and repeat all shard checks.
Model loading or runtime initialization runs out of GPU or host memory
Stop other GPU workloads and inspect nvidia-smi. Complete memory fit is unknown. Lowering context, changing precision or moving to different hardware creates a different configuration; do not report that as a successful run of this exact recipe.
Port 8000 is already used
Stop the conflicting service you own or choose another loopback port consistently in the start command, checks and SSH tunnel. Do not expose a public interface to work around a local port conflict.
A /v1 request returns 401
In the client terminal load the same private .vllm-api-key file used by the server and send Authorization: Bearer. Restart the server after deliberately rotating the key.

Stop the model

  1. 1. Stop the model process

    Press Ctrl-C in the terminal running vllm and wait for shutdown. Close any SSH forwarding process with Ctrl-C. Then check from the host terminal that the endpoint no longer answers. Stopping only the process does not stop an EC2 bill.

    Command
    curl --connect-timeout 3 http://127.0.0.1:8000/v1/models

    Expected result: The request cannot connect after the listener has stopped. Confirm no other process has taken the same port before assuming the host is idle.

Uninstall

  1. 1. Remove only this guide’s environment and files

    After stopping the server, preserve any needed outputs, then remove the dedicated $HOME/phi4-hosting directory you created, including its virtual environment, downloaded model and private key. Check the path before deleting; do not remove shared model caches or another project. Remove any copied client key as well. The host NVIDIA driver is not removed.

    Command
    deactivate
unset VLLM_API_KEY

    Expected result: The dedicated directory is removed when you choose to delete it; the shell no longer uses the virtual environment or service key. Shared host configuration remains separate.

Sources and verification

Sources checked Oct 15, 2026. Verification source

Compare this configuration
Cloud serverDocumented setupSources checked Oct 15, 2026

Run Phi-4 on AWS G6e through a private SSH tunnel

Set up Microsoft’s original BF16 Phi-4 on one AWS g6e.2xlarge with vLLM 0.31.0, an 8,192-token context, one active sequence and a private SSH tunnel. Includes pinned downloads, verification checks and cloud shutdown. This documented configuration has not been executed or benchmarked by OpenTools.

Hardware
AWS EC2 g6e.2xlarge (one NVIDIA L40S)
Runtime
vLLM 0.31.0+cu129 · CUDA
Model format
safetensors · BF16 · None; original BF16 weights
Workload
8,192 token context · 1 concurrent

Based on source documentation; this setup has not been executed by OpenTools. Memory: unknown. No compatible measurement available.

Read the complete setup guide

Before you start

  • AWS account, region with G6e capacity, sufficient GPU quota, an SSH key and permission to launch and stop the selected EC2 instance. Confirm current charges before launching.
  • Python 3.12 with virtual-environment support, curl, sha256sum, outbound HTTPS to the official package/model hosts, and a working CUDA-compatible NVIDIA driver.
  • The full repository download is about 29.33 GB. Allow additional disk for the virtual environment, package cache and runtime files; complete disk and RAM minima have not been measured.
  • A trusted single-user deployment: this guide binds vLLM to 127.0.0.1. Do not publish port 8000 or treat the API key as protection for every vLLM route.

Exact model and requirements

Artifact:
Microsoft / Phi-4 · checkpoint microsoft/phi-4
Publisher revision:
2db69c1c3e91a05d2c64a3185acfbaf36f744e25
Artifact digest:
The publisher exposes per-file hashes, not one digest for the complete multi-file artifact. Exact weight SHA256 and auxiliary Git blob hashes are listed below.
Download size:
27.31 GiB
Memory requirement (unknown):
Unknown — Complete GPU footprint has not been measured: model weights, KV cache, runtime allocations, compilation and workspace must all fit.
System RAM / disk:
Unknown / Unknown
Placement:
single device. One GPU; no tensor or pipeline sharding and no pooling of GPU and host RAM. No CPU or disk weight offload is enabled.
KV cache:
BF16 via --kv-cache-dtype auto with BF16 model dtype
Workload:
8,192 token context; Unknown; Unknown; concurrency 1; batch 1; cache unknown; repetitions Unknown.
Tokenizer:
microsoft/phi-4 · 2db69c1c3e91a05d2c64a3185acfbaf36f744e25
Sampling / reasoning:
Verification uses temperature 0 and max_tokens 64. The service uses --generation-config vllm; this is not a throughput protocol. Original Phi-4 checkpoint, not a Phi-4 reasoning variant; no special reasoning parser or budget is configured.
Runtime prerequisites:
Linux x86_64; Python 3.12; Working NVIDIA driver R580 or newer; CUDA 12.x applications remain supported by newer drivers through backward compatibility. Verify the selected environment with nvidia-smi and torch.cuda.is_available()..

Official artifact · License

The publisher declares MIT. Read the pinned LICENSE and retain its copyright and permission notice when redistributing covered material.

  • The six weight files total 29,319,042,992 bytes; that is a download measurement from publisher metadata, not the runtime GPU-memory requirement.
  • The selected L40S is labeled 48 GB by the vendor. Dedicated GPU memory and system RAM are different pools and are not summed.
  • No measured peak memory, completed-run evidence, tokens per second or latency result is supplied. A documented runtime architecture match does not prove this exact deployment fits.

One-time setup and hardware

Hardware purchase
Not applicable — This recipe rents EC2 capacity instead of purchasing a server; hourly compute is listed separately.
This recipe rents EC2 capacity instead of purchasing a server; hourly compute is listed separately. Tax: unknown.
Initial setup work
Unknown — Initial provisioning and setup time are not priced; no labour rate or duration has been measured.
Initial provisioning and setup time are not priced; no labour rate or duration has been measured. Tax: unknown.

Recurring costs

EC2 compute
Unknown — Choose a region and obtain a current g6e.2xlarge On-Demand quote before launching. No dated regional rate is asserted; running instances are billed even when idle.
Choose a region and obtain a current g6e.2xlarge On-Demand quote before launching. No dated regional rate is asserted; running instances are billed even when idle. SKU: g6e.2xlarge. Tax: unknown.
Persistent storage
Unknown — Root EBS and retained volumes can continue billing after compute stops; amount depends on selected capacity, type and region.
Root EBS and retained volumes can continue billing after compute stops; amount depends on selected capacity, type and region. Tax: unknown.
Data transfer
Unknown — Data transfer charges depend on direction, region and destination; no rate or traffic volume is assumed.
Data transfer charges depend on direction, region and destination; no rate or traffic volume is assumed. Tax: unknown.
Operations and maintenance
Unknown — Ongoing patching and operation effort is not measured or priced.
Ongoing patching and operation effort is not measured or priced. Tax: unknown.

Install the runtime

  1. 1. Select and connect to the EC2 host

    In the EC2 console, choose a region with G6e capacity and review its current On-Demand estimate and your GPU quota before launching. Select g6e.2xlarge and the AWS-published Deep Learning Base AMI with Single CUDA (Ubuntu 24.04) 20261002, x86_64. The dated AMI release notes list G6e and NVIDIA driver 595.91.07. Regional AMI IDs differ: verify the publisher and exact release name instead of using a random community image. Select a root EBS volume with at least 100 GB as a planning allowance for this walkthrough, not a tested minimum. Restrict inbound SSH to your own address; do not allow inbound TCP 8000. Connect using the SSH command shown in EC2 Connect; run the subsequent installation commands on that instance.

    Command
    nvidia-smi
python3.12 --version
df -h "$HOME"

    Expected result: nvidia-smi identifies one L40S and the selected AMI driver; Python reports 3.12; the chosen filesystem has adequate free space. Do not continue if GPU, architecture, image release or access controls differ.

  2. 2. Create an isolated environment and install pinned vLLM

    Run in a new shell on the GPU host. The exact CUDA 12.9 vLLM wheel is pinned by release and publisher-provided SHA256. Dependencies resolve inside this fresh environment; do not mix the AMI or host Python environment into it. The system CUDA toolkit version need not equal the bundled wheel runtime. If python3.12 -m venv is unavailable, install your OS vendor’s Python 3.12 venv package first.

    Command
    mkdir -p "$HOME/phi4-hosting"
cd "$HOME/phi4-hosting"
python3.12 -m venv .venv
source .venv/bin/activate
python -m pip install --upgrade pip
python -m pip install --extra-index-url https://download.pytorch.org/whl/cu129 'https://github.com/vllm-project/vllm/releases/download/v0.31.0/vllm-0.31.0%2Bcu129-cp38-abi3-manylinux_2_28_x86_64.whl#sha256=5176ab746f4e736c4a354a11d0620abf0a941de287cad937ac34039293ca6f5f'
python -m pip check
vllm --version
python -c 'import torch; print("CUDA available:", torch.cuda.is_available()); print("GPU:", torch.cuda.get_device_name(0) if torch.cuda.is_available() else "not detected")'

    Expected result: pip check reports no broken requirements; vllm reports version 0.31.0 (the wheel may include +cu129); CUDA available is True and the GPU is NVIDIA L40S. This is an expected check for the reader, not a result observed by OpenTools.

Download the model

  1. 1. Download the exact Microsoft snapshot

    Run after installing the runtime so its Hugging Face dependency is available. The full commit pins weights, configuration and tokenizer together; this does not follow the moving main branch. The official repository was ungated when checked. No model-provider API subscription is needed for this download.

    Command
    cd "$HOME/phi4-hosting"
source .venv/bin/activate
python - <<'PY'
from huggingface_hub import snapshot_download
snapshot_download(repo_id="microsoft/phi-4", revision="2db69c1c3e91a05d2c64a3185acfbaf36f744e25", local_dir="./model")
PY

    Expected result: The command completes and ./model contains config.json, tokenizer files, the safetensors index and all six weight shards from the pinned revision.

  2. 2. Verify every weight shard

    The SHA256 values come from the official repository’s file metadata. Stop if any shard is missing or mismatched; do not proceed with a mixed or partial download.

    Command
    cd "$HOME/phi4-hosting"
sha256sum --check <<'SHA256'
cafbe72210c39b19270cbfce27996afc4383c8a487087bcb91a6290d5ceb8f18  model/model-00001-of-00006.safetensors
9f87a5897162ed988cfb0fac13879634b4bab3da5a756b4f9dd2437ef8145e20  model/model-00002-of-00006.safetensors
19537df78a263aa0c0b2dc12dbce5908021e30a25e528699b08642c3999eeb05  model/model-00003-of-00006.safetensors
d4f17100a39c19d352f2e8108eef6fb975afb86808e50d78a075e162f1edfc93  model/model-00004-of-00006.safetensors
a72666b6001d76e7a9dc3ebe0d05a2c3316f546a8c6779c87343f296bdf82e50  model/model-00005-of-00006.safetensors
a9bfeeea354e31503d09a864e6cd5bd0f1230664065b6a2988b05e22b3228d77  model/model-00006-of-00006.safetensors
SHA256

    Expected result: All six listed safetensors files report OK.

Configure

  1. 1. Create a private local service key

    Run on the GPU host. This writes a fresh random key only when the file is absent, keeps it mode 600 and exports it without printing it. Treat the key file as private. vLLM API-key checks do not cover every route, so keep the loopback binding used below.

    Command
    cd "$HOME/phi4-hosting"
source .venv/bin/activate
umask 077
python - <<'PY'
import secrets
from pathlib import Path
key = Path(".vllm-api-key")
if not key.exists():
    key.write_text(secrets.token_urlsafe(32))
key.chmod(0o600)
PY
export VLLM_API_KEY="$(cat .vllm-api-key)"

    Expected result: A private .vllm-api-key file exists in the dedicated directory; no secret is printed. The shell environment contains VLLM_API_KEY.

  2. 2. Prepare the private SSH tunnel from your laptop

    After creating the key on the instance, run these commands in a separate laptop terminal. Enter the private-key path and the exact user@host target shown in EC2 Connect. The key is copied over SSH into a private local file; it is not printed. Leave the forwarding command running. Start vLLM in the instance terminal, then use another laptop terminal for clients, loading VLLM_API_KEY from $HOME/.phi4-client-api-key. Verify the SSH host fingerprint against your instance connection information.

    Command
    printf 'EC2 private-key path: '
IFS= read -r PHI4_SSH_KEY
printf 'EC2 SSH user@host: '
IFS= read -r PHI4_SSH_TARGET
umask 077
scp -i "$PHI4_SSH_KEY" "${PHI4_SSH_TARGET}:phi4-hosting/.vllm-api-key" "$HOME/.phi4-client-api-key"
chmod 600 "$HOME/.phi4-client-api-key"
ssh -i "$PHI4_SSH_KEY" -N -L 127.0.0.1:8000:127.0.0.1:8000 "$PHI4_SSH_TARGET"

    Expected result: SSH forwarding stays open. Once vLLM has started, your laptop loopback port 8000 forwards to the instance loopback; EC2 port 8000 remains closed to direct inbound access.

Start the model

  1. 1. Start the private Phi-4 endpoint

    Keep this foreground process in its own terminal. ./model is the verified pinned snapshot, so both weights and tokenizer are loaded from it. BF16 weights and automatic BF16 KV cache are used; there is no quantization or weight offload. One sequence is scheduled at a time. Startup can require compilation and workspace; do not infer success until the verification requests pass.

    Command
    cd "$HOME/phi4-hosting"
source .venv/bin/activate
export VLLM_API_KEY="$(cat .vllm-api-key)"
vllm serve ./model \
  --served-model-name phi4-local \
  --host 127.0.0.1 --port 8000 \
  --dtype bfloat16 --kv-cache-dtype auto \
  --max-model-len 8192 --max-num-seqs 1 \
  --tensor-parallel-size 1 --pipeline-parallel-size 1 \
  --generation-config vllm

    Expected result: The process remains running and announces an HTTP listener on 127.0.0.1:8000 after model initialization. If it exits, use the troubleshooting checks before making any success or performance claim.

Connect to your endpoint

http://127.0.0.1:8000/v1

The service listens only on the EC2 host loopback. On your laptop, set PHI4_SSH_KEY to your private key file and PHI4_SSH_TARGET to the user@host target from EC2 Connect. Run ssh -i "$PHI4_SSH_KEY" -N -L 127.0.0.1:8000:127.0.0.1:8000 "$PHI4_SSH_TARGET" and leave that tunnel open. Keep EC2 port 8000 closed. Local verification below may run in a second SSH session on the GPU host; laptop clients use the forwarded loopback endpoint and your service key.

Access: loopback. SSH controls remote access. VLLM_API_KEY authenticates /v1, /v2 and /inference paths only; other routes may remain unauthenticated. Loopback binding plus the restricted SSH tunnel is the intended boundary.

Verify the setup

  1. 1. Check the model identity

    Open a second terminal on the same GPU host. Load the local key without displaying it and call the authenticated models endpoint.

    Command
    cd "$HOME/phi4-hosting"
export VLLM_API_KEY="$(cat .vllm-api-key)"
curl --fail --silent --show-error http://127.0.0.1:8000/v1/models \
  -H "Authorization: Bearer $VLLM_API_KEY"

    Expected result: The JSON response contains the served model id phi4-local. The endpoint responds successfully only after startup completes.

  2. 2. Request a short response

    In that second terminal, send a small chat request. This checks response generation; it is not a benchmark and provides no throughput or latency guarantee.

    Command
    curl --fail --silent --show-error http://127.0.0.1:8000/v1/chat/completions \
  -H "Authorization: Bearer $VLLM_API_KEY" \
  -H 'Content-Type: application/json' \
  -d '{"model":"phi4-local","messages":[{"role":"user","content":"Explain what a GPU does in one sentence."}],"max_tokens":64,"temperature":0}'

    Expected result: A successful JSON response includes a nonempty choices[0].message.content. Wording varies; no exact answer or speed is guaranteed.

  3. 3. Verify the access boundary

    Check that the listener is only on 127.0.0.1:8000 and that the /v1 route rejects a missing key. For AWS, keep the security group closed to inbound 8000 and use the SSH tunnel for remote access.

    Command
    ss -ltn | grep '127.0.0.1:8000'
curl --silent --output /dev/null --write-out '%{http_code}\n' http://127.0.0.1:8000/v1/models

    Expected result: The listener is on loopback and the unauthenticated /v1/models request is denied (normally HTTP 401). This denial does not prove that every server path is authenticated.

Performance evidence

No compatible measurement available. Documented support and estimated memory do not establish response speed.

Limits and troubleshooting

  • This is a documented setup assembled from official model, hardware, runtime and cloud documentation. OpenTools has not executed these commands or verified runtime fit on the selected hardware.
  • The model supports a maximum 16,384-token context in its configuration; this recipe deliberately sets a smaller combined prompt-and-output context. Raising context or concurrency changes the configuration and memory demand.
  • Original Phi-4 can produce incorrect or unsafe output. It is not the Phi-4 mini or reasoning checkpoint. Apply the model publisher’s intended-use and limitation guidance.
  • Dependency packages installed with the pinned vLLM wheel are resolved at install time; capture pip freeze with your run report when reproducibility beyond the pinned runtime wheel is required.
  • Loopback is accessible to other users on the same machine. Use a trusted host. A publicly reachable production service needs a separate authenticated gateway and access design outside this single-user recipe.
nvidia-smi fails or PyTorch reports CUDA unavailable
Check the physical/virtual GPU and host driver before downloading weights. For the AWS path verify the exact supported G6e AMI. Use a fresh environment and the pinned CUDA 12.9 wheel; do not mix a pre-existing torch installation into it.
Installation or download runs out of disk space
Check the filesystem containing $HOME/phi4-hosting and package caches. The model download alone is about 29.33 GB; add space for the environment and runtime files. Resume the same pinned snapshot and repeat all shard checks.
Model loading or runtime initialization runs out of GPU or host memory
Stop other GPU workloads and inspect nvidia-smi. Complete memory fit is unknown. Lowering context, changing precision or moving to different hardware creates a different configuration; do not report that as a successful run of this exact recipe.
Port 8000 is already used
Stop the conflicting service you own or choose another loopback port consistently in the start command, checks and SSH tunnel. Do not expose a public interface to work around a local port conflict.
A /v1 request returns 401
In the client terminal load the same private .vllm-api-key file used by the server and send Authorization: Bearer. Restart the server after deliberately rotating the key.
The AWS tunnel cannot connect
Confirm the EC2 instance is running, SSH is limited to your current address, the private key and user@host target match EC2 Connect, and the vLLM listener is alive on the instance loopback. Keep inbound port 8000 closed.

Stop the model

  1. 1. Stop the model process

    Press Ctrl-C in the terminal running vllm and wait for shutdown. Close any SSH forwarding process with Ctrl-C. Then check from the host terminal that the endpoint no longer answers. Stopping only the process does not stop an EC2 bill.

    Command
    curl --connect-timeout 3 http://127.0.0.1:8000/v1/models

    Expected result: The request cannot connect after the listener has stopped. Confirm no other process has taken the same port before assuming the host is idle.

Uninstall

  1. 1. Remove only this guide’s environment and files

    After stopping the server, preserve any needed outputs, then remove the dedicated $HOME/phi4-hosting directory you created, including its virtual environment, downloaded model and private key. Check the path before deleting; do not remove shared model caches or another project. Remove any copied client key as well. The host NVIDIA driver is not removed.

    Command
    deactivate
unset VLLM_API_KEY

    Expected result: The dedicated directory is removed when you choose to delete it; the shell no longer uses the virtual environment or service key. Shared host configuration remains separate.

Shut down cloud resources

  1. 1. Stop EC2 compute and verify its state

    In EC2, select the exact instance used for this guide, choose Instance state → Stop instance, and wait for the state to become Stopped. Shutting down vLLM or closing SSH alone leaves a running instance billable. Stopping compute does not remove EBS storage charges.

    Expected result: The intended instance is Stopped, not merely stopping or idle; inspect retained EBS volumes and other resources for continuing charges.

  2. 2. Clean up resources when permanently finished

    Preserve needed data first. If the instance is no longer needed, terminate that exact instance in EC2 and verify Terminated. Review attached/preserved EBS volumes, snapshots and allocated addresses and remove only the resources you no longer need. Termination is irreversible; do not use it for a temporary pause.

    Expected result: The instance is Terminated and the billing/resource review accounts for any retained storage, addresses or other services. No zero-cost guarantee is implied.

Sources and verification

Sources checked Oct 15, 2026. Verification source

Compare this configuration

Benchmarks

Performance scores for Phi 4 across standard benchmarks.

GPQA DiamondArtificial Analysis
57.5%
AIME 2025Artificial Analysis
18%

Pricing

Token pricing for Phi 4 API usage.

Input Tokens

$0.07/M

per million tokens

Output Tokens

$0.14/M

per million tokens

Competing Models

Same pricing tier — direct alternatives to Phi 4

Efficiency
Hermes 4 70Bnous-research
$0.13/M / $0.40/MCompare
ERNIE 4.5 21B A3B Thinkingbaidu
$0.07/M / $0.28/MCompare
LFM2-24B-A2Bliquid-ai
$0.03/M / $0.12/MCompare
Olmo 3.1 32B Instructallenai
$0.20/M / $0.60/MCompare
Step 3.5 Flashstepfun
$0.10/M / $0.30/MCompare
Solar Pro 3upstage
5
13.9
$0.15/M / $0.60/MCompare
Granite 4.0 Microibm
$0.02/M / $0.11/MCompare
Hunyuan A13B Instructtencent
$0.14/M / $0.57/MCompare
DeepSeek V3.2deepseek
$0.26/M / $0.42/MCompare
Nemotron 3 Supernvidia
$0.09/M / $0.45/MCompare
ERNIE 4.5 21B A3Bbaidu
$0.07/M / $0.28/MCompare
Command R7B (12-2024)cohere
$0.04/M / $0.15/MCompare
Reka Flash 3reka
$0.10/M / $0.20/MCompare
Olmo 3 32B Thinkallenai
$0.15/M / $0.50/MCompare
DeepSeek V3.2 Expdeepseek
$0.27/M / $0.41/MCompare
Trinity Miniarcee-ai
$0.04/M / $0.15/MCompare
Hermes 3 70B Instructnous-research
$0.30/M / $0.30/MCompare
Llama Guard 3 8Bmeta
$0.48/M / $0.03/MCompare
Nemotron 3 Nano 30B A3Bnvidia
$0.05/M / $0.20/MCompare
Nova Micro 1.0amazon
$0.04/M / $0.14/MCompare
Command R (08-2024)cohere
$0.15/M / $0.60/MCompare
Leanstral 1.5mistral
Free / FreeCompare
North Mini Codecohere
Free / FreeCompare
GLM-5.3-Flashz-ai
74
456
$0.07/M / $0.25/MCompare
GPT-6 Lunaopenai
$0.10/M / $0.50/MCompare
Mercury 2.5inception
$0.04/M / $0.15/MCompare
Claude Haiku 5.5anthropic
$0.10/M / $0.50/MCompare
Microsoft logo
Microsoftenterprise
2 Models
View full profile
2 Models

Tools by Microsoft

Other AI tools from the same organization.

BitNet

BitNet

AI Infrastructure

BitNet - Official Inference Framework for 1-Bit LLMs

VibeVoice

VibeVoice

Audio AI

VibeVoice - Microsoft Open-Source Voice AI for Speech

AI Models by Microsoft

AI models from the same organization.

ModelContext WindowPrice (In / Out per M)
WizardLM-2 8x22BCurrent66K$0.62 / $0.62

MCP Servers by Microsoft

Connect this tool to AI assistants via the Model Context Protocol.

@microsoft/workiq

@microsoft/workiq

Uncategorized

official
Playwright MCP

Playwright MCP

Browser Automation

official32.0k

Related News

Latest coverage and updates.

GitLab Cuts 14% of Staff in AI Pivot Despite Record 264 Million Revenue

GitLab Cuts 14% of Staff in AI Pivot Despite Record 264 Million Revenue

Jun 8, 2026

AI Costs Spiral as Agentic Systems Burn 1000x More Tokens Than Chatbots

AI Costs Spiral as Agentic Systems Burn 1000x More Tokens Than Chatbots

Jun 1, 2026

Pope Leo Drops 42,000-Word AI Encyclical With Anthropic Co-Founder at His Side

Pope Leo Drops 42,000-Word AI Encyclical With Anthropic Co-Founder at His Side

May 27, 2026

Anthropic and OpenAI Race to Embed AI Agents on Wall Street

Anthropic and OpenAI Race to Embed AI Agents on Wall Street

May 23, 2026

OpenAI Launches Guaranteed Capacity to Lock In Compute Before IPO

OpenAI Launches Guaranteed Capacity to Lock In Compute Before IPO

May 20, 2026

Meta Lays Off 8,000 Staff May 20 as AI Capex Hits $145 Billion

Meta Lays Off 8,000 Staff May 20 as AI Capex Hits $145 Billion

May 18, 2026