Self-Hosted AI Operator Stack

What this collection sets up

This collection is for the operator side of a self-hosted AI stack. It starts with the local-first architecture, then separates responsibilities into model runtime selection, OpenAI-compatible routing, self-hosted MCP access, retrieval storage, model API packaging, container rebuilds, and image security checks.

The goal is not to make every workload fully offline. It is to make data paths, runtime choices, network exposure, credentials, logs, and rebuild behavior visible before agents or users depend on the stack.

Layers

1. Architecture and model runtime

• local-first-ai-dev-stack frames what stays on owned infrastructure and what may still call an external orchestrator or provider.
• ollama is the simplest local model runner for developer machines, delegation tasks, and offline fallback.
• llama-cpp covers lightweight GGUF-based inference for CPU, edge, and memory-constrained hosts.
• vllm covers higher-throughput GPU serving with OpenAI-compatible APIs, batching, structured outputs, and tool-calling support.

2. Gateway, tools, and retrieval

• litellm puts a model gateway in front of local and external providers so operators can manage routes, virtual keys, spending, and fallbacks.
• mcp-supergateway-hub exposes a fleet of stdio MCP servers over HTTP for private-network access from approved clients.
• chroma stores documents, embeddings, metadata, and retrieval indexes for local or self-hosted RAG and memory workflows.
• bentoml packages model inference code into model APIs and deployable service artifacts when the stack needs a production API surface.

3. Container operations

• docker-container-auto-rebuild helps rebuild affected containers after source or configuration changes.
• docker-image-security-scanner checks container images before they become part of the self-hosted AI environment.

Suggested order

Start by writing the local-first boundary and choosing the runtime for each workload: Ollama for simple local use, llama.cpp for compact GGUF serving, and vLLM for GPU-backed throughput. Add LiteLLM only after route and credential rules are clear. Bring up MCP Supergateway Hub and Chroma on a private network, then package production inference services with BentoML. Finish by adding container rebuild and image scanning hooks so stack changes are repeatable and reviewed.

Operator checklist

• {"task": "Boundary is written", "description": "Operators know which prompts, tools, models, embeddings, logs, and fallbacks are allowed to leave owned infrastructure"}
• {"task": "Runtime fits hardware", "description": "CPU, RAM, GPU, VRAM, disk, context length, and concurrency match the selected model runtimes"}
• {"task": "Endpoints are private", "description": "Model, MCP, retrieval, and API services have authentication, TLS or private-network controls, and rate limits"}
• {"task": "Credentials are scoped", "description": "Model gateway keys, registry tokens, provider fallbacks, and MCP secrets are rotated and least-privilege"}
• {"task": "Containers are reviewable", "description": "Rebuilds, image scans, logs, and rollbacks are part of the normal operator workflow"}
• {"task": "Retention is explicit", "description": "Prompts, outputs, embeddings, traces, scanner reports, and backups have an owner and deletion policy"}

Source and references

• Ollama documentation: https://docs.ollama.com/
• llama.cpp documentation: https://github.com/ggml-org/llama.cpp/tree/master/docs
• vLLM documentation: https://docs.vllm.ai/en/stable/
• LiteLLM documentation: https://docs.litellm.ai/docs/
• Model Context Protocol documentation: https://modelcontextprotocol.io/docs/getting-started/intro
• MCP Supergateway Hub repository: https://github.com/dpdanpittman/mcp-supergateway-hub
• Chroma documentation: https://docs.trychroma.com/docs/overview/introduction
• BentoML documentation: https://docs.bentoml.com/en/latest/
• Docker Compose documentation: https://docs.docker.com/compose/
• Trivy documentation: https://aquasecurity.github.io/trivy/

Duplicate check

Checked existing collections, guides, tools, skills, hooks, open PRs, closed PRs, and issue history for self-hosted-ai-operator-stack, self-hosted AI operator, local-first AI stack, Ollama, llama.cpp, vLLM, LiteLLM, MCP Supergateway Hub, Chroma, BentoML, Docker rebuild hooks, and image security scanning. local-first-ai-dev-stack is a guide for one local-first developer architecture. agent-operator-growth-master-pack is a broad product-operator bundle covering review, release, growth, and automation. This collection is narrower and operational: it bundles the runtime, gateway, MCP, retrieval, model API, rebuild, and scan entries needed to run self-hosted AI services.

Disclosure

Editorial collection. No paid placement or affiliate link is used.