Open-weight model gateway
Run compatible Safetensors, native MLX, and quantized GGUF paths across CPU, GPU, and Apple Silicon. Keep one application API, model lifecycle, and cluster control plane across all three.
01 / Open-model platform
Open-weight models differ in format, quantization, architecture, and hardware target. Gateway unifies their operational layer without forcing every model through one execution engine.
Safetensors ecosystem
Use compatible Safetensors across the MLX and general-purpose engines while Gateway keeps architecture support, tokenizers, model rules, and capabilities explicit.
Native Apple Silicon
Gateway includes a native MLX provider for compatible Safetensors models on Apple Silicon, exposed through the same discovery and lifecycle controls.
Broad quantized support
Gateway supports quantized GGUF models across CPU, Metal, and CUDA targets with model-specific rules, shared APIs, and the same operational lifecycle.
02 / Where it fits
Gateway turns open models into application capabilities. Use them for assistants, private knowledge, and product features instead of stopping at a standalone model runner.
Private assistant
Build internal assistants for drafting, summarization, question answering, and tool-driven tasks on infrastructure you operate.
Private knowledge
Create an internal retrieval pipeline with local embeddings and reranking, then generate answers against the retrieved context.
Product integration
Use familiar API contracts for assistants, automation, and developer tools without building a separate integration for every model runtime.
03 / Local runtimes
Choose compatible Safetensors through the general-purpose engine, MLX on Apple Silicon, or quantized GGUF across supported hardware. Gateway keeps each runtime explicit, provides one application-facing entry point, and adds lifecycle controls where the runtime is managed by Gateway.
Safetensors → MLX / general-purpose engine · GGUF → llama.cpp
CPU · Apple Silicon / Metal · NVIDIA / CUDA
The runtime is never guessed Use a provider field or provider::model prefix. A mismatched combination is rejected instead of silently running somewhere else.
One lifecycle for managed runtimes Discover, inspect, pull, load, unload, and observe every managed model — GGUF, MLX, and the general-purpose engine — through one operating flow. A checkpoint you place under an engine's model root yourself is detected without a download.
A stable, capability-aware API Use Responses, Chat Completions, and Messages across compatible paths. Embeddings, rerank, and specialized endpoints remain explicit capabilities of the selected model and runtime.
curl http://localhost:10037/v1/responses \
-H "Content-Type: application/json" \
-d '{
"provider": "gguf",
"model": "<model-id>",
"input": "Hello"
}'{
"model": "gguf::<model-id>",
"input": "Hello"
}04 / Local AI patterns
These are not generic capability labels. Each pattern maps to request and runtime paths implemented in Gateway.
Answer questions, summarize, draft, and reason over user input while keeping the inference path inside your environment.
Call tools, limit parallel tool execution, and return validated structured results for downstream automation.
Use raw completions and fill-in-the-middle requests with models that expose the required FIM tokens.
Create embeddings, rerank retrieved documents, and run token classification without sending the source corpus to an external API.
Generate JSON that follows a schema, constrain decoding with a grammar, and classify tokens for machine-readable local workflows.
Build local translation, transcription, speech translation, and text-to-speech flows with supported models and builds.
Each capability is checked against the model manifest, runtime, platform, and active build. Unsupported inputs return an error rather than being silently ignored.
05 / Model lifecycle
Local AI becomes useful when models can be found, validated, loaded, observed, and released repeatedly. Gateway provides that lifecycle for every runtime it manages — GGUF, MLX, and the general-purpose engine.
List local files and catalog entries across configured search roots.
Check format, size, compatibility, capabilities, and manifest rules before inference.
Download catalog models as tracked jobs with status, recent history, and cancellation.
Preload a model, apply runtime configuration, and make it available to inference endpoints.
Unload idle models automatically or explicitly, and remove downloaded models when no longer needed.
Gateway Hub
Search the Gateway model manifest and inspect the pull command and runtime metadata for each entry.
06 / Distributed
When one device is not enough, Gateway keeps the application contract intact while its control plane represents placement, deployment, topology, probing, and diagnostics across coordinated nodes.
07 / Operations
A useful local model needs more than a successful first response. Gateway provides testing, visibility, access control, and lifecycle operations for continued use.
PLAYGROUND
Switch endpoint, provider, model, task, reasoning, sampling, and streaming settings, then export runnable REST, TypeScript, or Python code.
OBSERVABILITY
Request logs, service status, loaded models, active jobs, GPU information, server timing, and Prometheus metrics are available directly.
SECURITY
TLS, scoped Gateway access tokens, and provider credentials are handled separately so control-plane secrets are not forwarded upstream.
CLI
Inspect status, discover models, pull, load, run, stop jobs, unload, configure auto-unload, and shut down gracefully from the gateway CLI.
08 / Why Gateway
Launching one model is the beginning. Gateway keeps model choice open while adding the shared API, lifecycle, observability, and cluster controls needed to keep open models useful.
Move across compatible Safetensors, native MLX, and quantized GGUF paths according to the model, hardware, and deployment you choose.
Use the native MLX path and its model lifecycle controls without creating a separate application integration for Apple Silicon.
Move from one machine to coordinated nodes while keeping model placement and distributed execution behind the Gateway contract.
Inspect capabilities, manage downloads and loaded state, observe requests and resources, secure access, and diagnose deployment health from one operating layer.
09 / Start
Choose a model for chat, extraction, coding, retrieval, or speech; load it into the matching runtime; then call it through the API shape your application already understands.
# Start the server
gateway serve
# Check service and loaded models
gateway ps
# Run against an explicit target
gateway run gguf:<model-id> -p "Hello"Inomina Gateway
Move across formats, runtimes, hardware, and cluster sizes without rebuilding the operating layer around every model.