Loading catalog…
Product briefing
local-llm-server
ProductionData & Intelligence
A production-ready local model serving platform with an OpenAI-compatible API, orchestration endpoints for model lifecycle management, deep performance telemetry, and an optional Vector-Tester companion UI for staged testing and crash-log capture.
API: Available
Endpoints
6
Key features
6
Stack components
8
Controls noted
5
What changed
May 15, 2026: Added governance-aligned copy, compare tooling support, and transparency notes.
Key Features
- 1Model registry with load and unload orchestration
- 2OpenAI-compatible and native streaming inference endpoints
- 3Real-time GPU, CPU, TTFT, and throughput telemetry
- 4Efficiency mode for lower-overhead inference logging
- 5Cache inspection and cleanup workflows
- 6Vector-Tester companion UI for staged model tests and Hugging Face metadata capture
API Endpoints
| Method | Path | Description |
|---|---|---|
| GET | /api/models | List registered models and metadata |
| POST | /api/orchestrate/load | Load a specific model into VRAM |
| POST | /api/orchestrate/unload | Unload current model to free VRAM |
| POST | /api/generate/stream | Run streaming inference |
| GET | /api/status | Get current server and model status |
| GET | /api/cache/stats | Inspect cache usage and cleanup opportunities |
Usage Example
python
import requests
# Example interaction
response = requests.get(
url="https://api.arcore.internal/v1/models",
headers={"Authorization": "Bearer <token>"}
)
print(response.json())Recommended next products
Tech Stack
PythonFastAPISQLModelSQLiteDockerROCm/CUDAllama.cppNext.js
Authentication
- •**Header:** `X-API-Key: <key>` (when enabled)
- •**Docs:** Swagger and ReDoc are exposed locally for operator use
- •**Scopes:** Access is controlled per deployment; the platform is designed for private or air-gapped environments.
Compliance & Security
Compliance
- ✓Network: Air-gap capable for on-prem or offline deployments
- ✓Access: API key authentication with local-only documentation endpoints
- ✓Data Privacy: No external data egress during inference
- ✓Operations: Model validation, cache inspection, and staged load testing are built in
Security
- ✓Access: API key authentication with local-only documentation endpoints
Coming Soon
3 plannedContent Filtering Policies
Target: Q1 2025
Per-User Resource Quotas
Target: Q2 2025
RBAC Integration for Model Access
Target: Verification Required
Related Products
Arcore Maestro
Arcore Maestro is an AI workflow orchestration platform that routes tasks through LangGraph-based workflows, local or remote LLMs, and sandboxed worker services. It provides a unified control plane for defining, executing, and monitoring multi-step AI pipelines across the Arcore ecosystem.
Graph-based workflow orchestration with conditional branching
CRUD management for services, models, workflows, graphs, and templates
PythonFastAPIReact
6 endpointsOpen
Chapterize
An intelligent PDF processing platform that converts unstructured books and documents into structured chapters with rich HTML, text, and images. Chapterize combines a Django + React web application with a Python CLI toolkit for batch processing and downstream RAG-friendly exports.
TOC-first chapter detection with regex fallback
Layout-aware extraction of text, images, and tabular regions
PythonDjangoReact
5 endpointsOpen
Arcore Career
A comprehensive career management platform that models a user's professional identity as a queryable knowledge graph. ArcoreCareer combines application tracking, resume and cover-letter generation, interview prep, skill intelligence, and governed LLM-assisted fact extraction in a single local application.
130+ entity career knowledge graph
Full-lifecycle job application tracking and pipeline analytics
Next.js 14TypeScriptPrisma
6 endpointsOpen