Inference configs for local LLMs
Ready-to-run settings to run a model on your own machine: Ollama Modelfile, vLLM command, llama.cpp.
- [8hyjqo] llama.cpp server for a GGUF model | inference | en | MIT | https://contextetech.com/en/inference/llama-cpp-server-gguf-8hyjqo - [1rcm91j] Mistral 7B on a 16 GB Mac with Ollama | inference | en | MIT | https://contextetech.com/en/inference/mistral-7b-ollama-mac-16gb-1rcm91j - [l4t26x] Qwen2.5 7B with vLLM on a 24 GB GPU | inference | en | MIT | https://contextetech.com/en/inference/qwen2-5-7b-vllm-24gb-gpu-l4t26x
contextetech
llama.cpp server for a GGUF model
Run any GGUF model with the llama.cpp server: OpenAI-compatible API on port 8080, works without a GPU too.
contextetech
Mistral 7B on a 16 GB Mac with Ollama
Ollama Modelfile to run Mistral 7B Instruct locally on a 16 GB Mac: Q4 quantization, 8k context, concise answers in English.
contextetech
Qwen2.5 7B with vLLM on a 24 GB GPU
Serve Qwen2.5 7B Instruct with vLLM on a 24 GB GPU: OpenAI-compatible API, 8k context, a single command.
What is an inference config?
An inference config is the set of settings needed to run a model on your own machine: the runtime (Ollama, vLLM, llama.cpp), quantization, context size and launch command.
Guide: Inference configs →