llminferencearchitecture

Local LLM Inference Architecture — From Silicon to Stack

Deepak Gupta · ·19 min read
Local LLM Inference Architecture — From Silicon to Stack

A unified stack for running, routing, and observing LLMs — currently deployed on a laptop with RTX 3050 Ti, with planned expansions to K8s GPU nodes and Apple Silicon clusters.

✓ Ollama Running ✓ Observability Stack Planned: K8s GPU Planned: Mac Mini MLX

Current Deployment Status Live

✓ Running: Ollama

  • Endpoint: localhost:11434
  • qwen2.5-coder:1.5b — 986MB, 50 layers GPU
  • qwen2.5-coder:7b — 4.7GB, CPU-only
  • nomic-embed-text — 274MB embeddings
  • GPU: RTX 3050 Ti, 4GB VRAM

✓ Running: Observability

  • Prometheus: :9090 (metrics)
  • Grafana: :3001 (dashboards)
  • OTEL Collector: :4317/:4318 (traces)
  • Loki: :3100 (logs)
  • Tempo: :3200 (traces)

Planned: Expansion

  • K8s GPU Node: Add NVIDIA GPU node
  • vLLM: Tensor parallel inference
  • Mac Mini: MLX cluster (TBD)
  • LiteLLM: Multi-backend routing

Quick Start Commands

# Start Ollama
ollama serve &

# Start observability stack
cd ~/Documents/git/local-llm/observability && docker compose up -d

# Verify everything is running
curl http://localhost:11434/v1/models # Ollama
curl http://localhost:9090/-/healthy # Prometheus
curl http://localhost:3001/api/health # Grafana

The Complete Stack Architecture

End-to-End Data Flow

VS Code + Champ Routing Inference Backends Observability Champ Agent Multi-agent orchestrator LiteLLM :4000 fallback + cost Laptop (RTX 3050 Ti) Ollama · Qwen 7B (CPU) + 1.5B (GPU) nomic-embed-text AnalyticsExporter OTLP JSON → :4318 llm-d Router KV-cache · EPP + Envoy K8s Cluster (GPU Nodes) vLLM · tensor parallel · P/D disaggregation KV offload GPU→CPU→SSD Prefix-aware scheduling RAG Indexer AST chunking · vector search MLX Launch TB5 / JACCL RDMA Mac Mini Cluster (Apple Silicon) MLX · tensor/pipeline/expert parallel Unified memory · OpenAI-compat server AuditLog · SHA-256 chain .champ/audit.log OTel Collector OTLP receiver → batch → tail sampling VictoriaMetrics MetricsQL · :8428 VictoriaTraces Jaeger · :10428 VictoriaLogs LogsQL · :9428 vmalert Alert rules Grafana :3000 LLM Ops · RAG · MCP · GPU · User dashboards Opik :5173 Hallucination · RAG quality · Prompt A/B ClickHouse + MySQL backend dcgm-exporter :9400 — GPU util · VRAM · temp

Parallelism Strategies How Models Split

Tensor Parallelism

Split each layer by width. Every node holds part of every layer. All nodes process the same token simultaneously.

All-reduce per layer Low latency Needs fast interconnect

Pipeline Parallelism

Split model by depth. Each node holds a contiguous slice of layers. Activations flow node-to-node in a pipeline.

P2P between stages Higher latency Less bandwidth

Expert Parallelism

MoE models only. Different experts live on different nodes. Router dispatches tokens. Only active experts communicate.

MoE-specific Scales well to 4 nodes Llama 4 Maverick

How Parallelism Splits a Transformer Block

Tensor Parallelism Attention Heads Node 0 Node 1 FFN / MLP All-Reduce Next Attention Heads Next FFN / MLP All-Reduce Pipeline Parallelism Layers 0–7 Node 0 Layers 8–15 Node 1 Layers 16–23 Node 2 Activations flow sequentially Higher latency, less bandwidth Expert Parallelism (MoE) Router Expert 0 Node 0 Expert 1 Node 1 Expert 2 Node 2 Only active experts communicate per token Active expert (green dot)

KV-Cache Architecture llm-d

Hierarchical KV-Cache with Prefix-Aware Routing

GPU HBM Fastest · ~80 GB · Hot blocks Tier 0 · ns access CPU DRAM Medium · ~64 GB · Warm blocks Tier 1 · μs access Shared SSD / FS Largest · TB-scale · Cold blocks Tier 2 · ms access · cross-replica Incoming Request "Explain the refund policy for order #1234..." Tokenize + Hash prefix Query KV-Block Index EPP Scores Pods → Route to warmest cache Score 9 Score 5 Score 1 vLLM Pod Fleet Pod A (warm) prefix cached ✓ Pod B (partial) prefix 60% Pod C (cold) no cache match Pod D (partial) prefix 40% KV-Events (Write Path) vLLM emits BlockStored / BlockRemoved via ZMQ Event Subscriber → KV-Block Index (in-memory LRU) PrefixStore caches tokenized prompt prefixes Performance Impact Without KV routing TTFT: 2,850ms With KV routing TTFT: 340ms (88% faster) Cache hit rate 87.4%

MLX Mac Cluster Apple Silicon

Thunderbolt 5 Mesh — JACCL RDMA

Mac Mini 0 Coordinator Rank 0 · mlx.launch Mac Mini 1 Rank 1 · Worker TB5 RDMA peer Mac Mini 2 Rank 2 · Worker TB5 RDMA peer Mac Mini 3 Rank 3 · Worker TB5 RDMA peer TB5 80 Gbps TB5 80 Gbps TB5 TB5 Model Tensor Parallelism across 4 Macs Layer 0 (heads split across 4 ranks) Layer 1 (heads split across 4 ranks) …

MLX Distributed Stack

  • JACCL — RDMA over Thunderbolt 5, 80 Gbps direct memory
  • mlx.distributed — all_reduce, all_gather, send/recv primitives
  • mlx-lm — sharded_load, stream_generate, —pipeline flag
  • mlx.launch — SSH orchestration from your MacBook

Scaling Numbers

SetupTok/sTTFTMax Model (4-bit)
2x Mac Mini M4 16GB~12~700ms~24 GB
4x Mac Mini M4 16GB~20~500ms~48 GB
2x Mac Studio M5 Ultra~14~740ms~400B MoE
4x Mac Studio M5 Ultra~35~400ms1T params

Prefill/Decode Disaggregation llm-d on K8s

Separated Prefill and Decode Stages

Client Champ / API Envoy Gateway ext_proc → EPP EPP Scheduler KV-cache aware + load-aware Prefill Workers P1 P2 P3 Compute-bound · wide EP Decode Workers D1 D2 Latency-bound · more TP KV transfer (NIXL/UCCL) Shared KV Storage GPU → CPU → SSD Cross-replica reuse Hybrid model HMA-aware Scale-to-zero capable P/D vs Aggregated E2E latency: −47% | ITL: −67% at 45 QPS, 20:1 ISL:OSL ratio

Observability Stack Metrics · Traces · Logs

Signal Flow — From Code to Dashboards

Sources Ollama /metrics OTEL SDK spans App logs System metrics OTel Collector OTLP receiver Batch processor Memory limiter Export by signal: metrics → Prometheus traces → Tempo logs → Loki Prometheus :8889 Prometheus TSDB · 7d retention :9090 Tempo Trace storage · :3200 Loki Log aggregation · :3100 Grafana Dashboards LLM Inference Dashboard GPU Metrics Trace Explorer Log Analytics Alert Rules Alert Rules Service down → critical Error rate > 5% → warn Latency p99 > 10s → warn GPU temp > 85°C → crit

docker-compose.yml

# Location:
~/Documents/git/local-llm/observability/

# Services:
prometheus:9090
grafana:3001
otel-collector:4317/:4318
loki:3100
tempo:3200

Grafana Credentials

# URL:
http://localhost:3001

# Username: admin
# Password: admin

# Pre-provisioned datasources:
Prometheus (default)
Tempo
Loki
OTEL (collector metrics)

Key Metrics What to Monitor

LLM Performance

MetricType
champ_llm_ttft_secondsHistogram
champ_llm_request_duration_secondsHistogram
champ_llm_tokens_per_secondGauge
champ_llm_input_tokens_totalCounter
champ_llm_output_tokens_totalCounter
champ_llm_cost_usd_totalCounter
champ_llm_errors_totalCounter

Agent & Tool

MetricType
champ_agent_step_duration_secondsHistogram
champ_tool_call_duration_secondsHistogram
champ_mcp_tool_call_duration_secondsHistogram
champ_rag_query_duration_secondsHistogram
champ_rag_result_countHistogram
champ_hallucination_scoreHistogram

Infrastructure

MetricSource
DCGM_FI_DEV_GPU_UTILdcgm-exporter
DCGM_FI_DEV_FB_USEDdcgm-exporter
DCGM_FI_DEV_TEMP_GPUdcgm-exporter
vllm:gpu_cache_usage_percvLLM /metrics
vllm:num_requests_runningvLLM /metrics
kv_cache_hit_ratellm-d EPP

Quick Reference Config Cheatsheet

Champ → LiteLLM (Single URL)

Champ sees one OpenAI-compatible endpoint. LiteLLM routes to everything.

champ.provider: openai-compatible
champ.openaiCompatible.baseUrl: http://localhost:4000
champ.openaiCompatible.model: coding
champ.openaiCompatible.apiKey: sk-champ
champ.autocomplete.model: fast
champ.indexing.embeddingProvider: ollama

LiteLLM → Model Fleet

One config file, every backend behind one port.

model_list:
  - model_name: coding
    litellm_params:
      model: ollama/qwen2.5-coder:7b
      api_base: http://localhost:11434
fallbacks: [{coding: [fast]}]

Enjoyed this post?

Get the next one in your inbox — only when I ship something worth reading.

Newsletter form not configured.

Or follow on Substack for the newsletter.

Comments via GitHub Discussions

Comments not configured. Set GISCUS env vars to enable.

Related Posts

mcp

Agentic AI, MCP & Gateways — Complete Visual Guide

Strip away the hype and an AI agent is one simple thing: a language model running in a loop, calling tools, and reading the results until a goal is reached. A chatbot answers once and stops. An agent acts — it decides its own next step based on what just happened in the environment.