Infrastructure: inference configs and AI stacks
Run a model on your own machine.
- [8hyjqo] llama.cpp server for a GGUF model | inference | en | MIT | https://contextetech.com/en/inference/llama-cpp-server-gguf-8hyjqo - [10wkyv1] Local automations: n8n and Ollama | stack | en | MIT | https://contextetech.com/en/stacks/local-automations-n8n-ollama-10wkyv1 - [1dabn54] Minimal private AI chat: Ollama and Open WebUI | stack | en | MIT | https://contextetech.com/en/stacks/minimal-private-ai-chat-1dabn54 - [1rcm91j] Mistral 7B on a 16 GB Mac with Ollama | inference | en | MIT | https://contextetech.com/en/inference/mistral-7b-ollama-mac-16gb-1rcm91j - [16vk3g8] Private chat on your documents: Ollama, Qdrant, Open WebUI | stack | en | MIT | https://contextetech.com/en/stacks/private-chat-on-your-documents-16vk3g8 - [l4t26x] Qwen2.5 7B with vLLM on a 24 GB GPU | inference | en | MIT | https://contextetech.com/en/inference/qwen2-5-7b-vllm-24gb-gpu-l4t26x
llama.cpp server for a GGUF model
Run any GGUF model with the llama.cpp server: OpenAI-compatible API on port 8080, works without a GPU too.
Local automations: n8n and Ollama
docker-compose stack to automate with a local AI: n8n (workflows) and Ollama (model), without sending data to an external API.
Minimal private AI chat: Ollama and Open WebUI
The simplest stack for a private AI chat: Ollama and Open WebUI with docker-compose, interface on port 3000.
Mistral 7B on a 16 GB Mac with Ollama
Ollama Modelfile to run Mistral 7B Instruct locally on a 16 GB Mac: Q4 quantization, 8k context, concise answers in English.
Private chat on your documents: Ollama, Qdrant, Open WebUI
docker-compose stack for a private chat on your documents: Ollama (model), Qdrant (vector database) and Open WebUI (interface).
Qwen2.5 7B with vLLM on a 24 GB GPU
Serve Qwen2.5 7B Instruct with vLLM on a 24 GB GPU: OpenAI-compatible API, 8k context, a single command.
What is AI model deployment?
Deployment means installing and configuring a model for use in an application, on a server or on a local machine. This family brings together resources to run the model, receive requests and generate responses with the available hardware.