MLX-OC — How LLM Monitoring Works

MLX

Component by component: what is measured, which macOS tool or Python library produces it, how it travels, and where it lands.

1Data Path

The One-Sentence Version

Two independent paths carry everything. Pull: five Python agents expose /metrics and VictoriaMetrics scrapes them every 10 seconds. Push: the proxy and the log tailer ship OpenTelemetry over HTTP to a collector, which writes metrics into the same store. Nothing in cluster/ ever pushes a hardware metric — the hardware exporter is pull-only by design, so those series exist exactly once.

Data Path Overview — Pull & Push Lanes

The Eight Components

Two Mac Minis (rank 0 and rank 1, joined by a Thunderbolt ring) plus one management node. Every row below is a real process with a real port.

Component	Port	Runs On	What It Owns	Transport
mlx_metrics_proxy.py	8080	Rank 0	The OpenAI-compatible front door. Every LLM RED, quality and confidence metric is born here.	Pull and Push
mlx_lm.server	8081	Rank 0 (+ ring)	The actual inference engine. Exposes no stats endpoint — which is why a log-scraping agent exists.	None
mlx_hw_telemetry.py	9102	Both nodes	GPU/ANE/CPU power, thermals, memory, processes, Thunderbolt counters. Pull-only, deliberately.	Pull
mlx_kv_cache_agent.py	9104	Rank 0	KV cache occupancy and context budget — parsed out of server.log text.	Pull
mlx_server_supervisor.py	9105	Rank 0	Process truth: alive, ready, model loaded, restarts, crash reason, backoff.	Pull
mlx_server_log_tailer.py	9106	Rank 0	Tails server.log, ships lines as OTLP logs, counts shipped and dropped.	Push
victoria-metrics	8428	Rank 0	Scrapes everything every 10 s and receives remote_write. Single TSDB, 30-day retention.	—
vmalert + alertmanager	8880 9093	Rank 0	46 alert rules evaluated against VM; routed, grouped and inhibited, then POSTed to two webhooks.	—

Why Two Paths At All

The split is a deliberate design decision, and the proxy’s own docstring states it.

Hardware Is Pull-Only

mlx_hw_* series exist exactly once in VictoriaMetrics because the hardware exporter has no OTLP exporter at all — mlx_hw_telemetry.py:17-19 says so directly. If it also pushed, every gauge would be double-written.

LLM Metrics Are Local

The proxy serves its own /metrics on the same port that serves inference (mlx_metrics_proxy.py:915-921). One listener, two purposes — scraping costs nothing and keeps the counters exactly where the requests already are.

Opik Bypasses The Collector

Opik receives traces directly from the proxy via a second BatchSpanProcessor, not through otel-collector. There is no opik exporter anywhere in observability/.

Push Toward Nodes Only

Nothing runs on the Macs to be scraped for push purposes. VictoriaMetrics scrapes outward, Opik is called outward, and rank 0 is the only host that needs to reach anything.

2Metric Collection

The Proxy Is The Whole LLM Observability Story

Forty metric families live in one 1732-line file. This is the instrumentation lifecycle of a single chat completion, and which metric families each phase feeds.

Inside The Proxy — One Request, Six Instrumentation Phases

How Logprobs Are Obtained — Three Sources

The confidence metrics only exist if the proxy can get logprobs out of the inference engine. mlx_lm.server returns logprobs only for non-streaming responses, so the proxy manufactures a scored path three different ways and records which one was used in mlx_confidence_scored_total{source}.

Client

The caller asked for logprobs itself. Cheapest path — the proxy just reads what came back and passes it through.

Injected

A non-streaming request that did not ask. The proxy adds logprobs: 3 (MLX_LOGPROBS), scores it, then strips it back out so the client never sees it.

Destreamed

A streaming request, re-run non-streamed so it can be scored. Costs one extra generation and the client gets its whole answer at once. Sampled at MLX_LOGPROBS_STREAM_SAMPLE (default 0.05).

Cost you should know about: destreaming doubles the work for the sampled fraction of streaming traffic. MLXConfidenceScoringStopped exists specifically to tell you when this pipeline silently dies — and its own description notes that “the confidence alerts above are blind until this clears.”

Hardware Collection — Ten Commands, Two Cadences

The hardware exporter is a plain prometheus_client registry behind a ThreadingHTTPServer. Two sampling threads run at different speeds because powermetrics is a ~1 second blocking call. Ten distinct commands are shelled out, across nine boxes below — ifconfig also runs, to auto-detect which interface carries the ring.

mlx_hw_telemetry.py — Command → Cadence → Exported Series

Passwordless sudo is not optional here. Two powermetrics invocations drive 7 of the 31 hardware gauges — CPU temperature (--samplers smc) plus GPU utilization, GPU frequency, and CPU/GPU/ANE/package power (--samplers cpu_power,gpu_power). Without sudo -n working they all degrade to NaN, and every power and throttling alert goes blind. The -n flag makes them fail fast rather than hang.

The Odd One Out — KV Cache By Log Parsing

mlx_lm.server has no stats endpoint, so mlx_kv_cache_agent.py reads the server’s own log file. It seeks to the last 1,000,000 bytes, discards the partial first line, and regex-matches the cache banner.

KV

Prompt Cache: 4 sequences, 0.03 GB

Two regexes recover cache state — one for the total, one for the per-role breakdown (assistant / user / system). GB is multiplied by 1e9 to get bytes. Only the fixed four role gauges are published; anything else is zeroed.

cluster/mlx_kv_cache_agent.py:46-47, 73-96

Utilization Is Derived, Not Reported

mlx_kv_cache_utilization is computed as total_bytes / (10 × kv_bytes_for_max_context). The 10 is a hard-coded literal matching the server’s --prompt-cache-size 4 and LRUPromptCache default — it is not read from config.

A Metric Name Lives In Two Places

mlx_context_length_max_tokens and mlx_kv_bytes_per_token are exported by both the proxy (labelled model) and the KV agent (unlabelled). Two series sets, one name. Grafana papers over it with max(...).

3Transport & Storage

The Collector — One Receiver, One Processor, Three Pipelines

otel-collector-config.yaml is only 49 lines. Everything flows through a single batch processor, then fans out to three exporters depending on signal type.

otelcol-config.yaml — Pipeline Graph

Every Scrape Job

Generated by setup.sh from vm-scrape.tmpl.yml via sed — two placeholders, __MLX_IP__ and __RANK1_IP__. The output vm-scrape.yml is gitignored.

Job	Targets	Interval	Static Labels	Measures
mlx-proxy	<MLX_IP>:8080	10 s	service=mlx-proxy, node=rank0	Requests, tokens, TTFT, generation time, token rate, finish reason, tool calls, cache ratio, hallucination risk, confidence
mlx-hw	<MLX_IP>:9102 and <RANK1_IP>:9102	10 s	service=mlx-hw, node=rank0 / rank1	Power, thermals, GPU utilization and frequency, memory, processes, Thunderbolt ring counters
mlx-kv	<MLX_IP>:9104	10 s	service=mlx-kv, node=rank0	KV cache sequences and bytes by role, cache utilization, context budget
mlx-supervisor	<MLX_IP>:9105	10 s	service=mlx-supervisor, node=rank0	Server state, up, ready, model loaded, restarts, crash reason, backoff, health-check outcomes
mlx-logtailer	<MLX_IP>:9106	10 s	service=mlx-logtailer, node=rank0	Tailer health, lines read, shipped, dropped by reason, file offset, queue depth
sglang	<MLX_IP>:30000, <RANK1_IP>:30000	15 s	service=sglang, node=rank0 / rank1, provider=sglang	Experimental second inference backend. Marked unverified in the config header — no ring or tensor-parallel backend on Apple Silicon.

Alert Routing

vmalert evaluates 46 rules against VictoriaMetrics and POSTs to Alertmanager. The root route groups by alertname, node, model, waits 30 s, repeats every 4 h, and splits on severity.

Two Receivers, Both Webhooks

critical → oncall receiver. Everything else → default. Both POST to host.containers.internal:9000 with send_resolved: true. No email, no Slack, no PagerDuty.

Two Inhibition Rules

MLXProxyDown suppresses MLXUpstreamDown, MLXThroughputDrop and MLXInferenceStalled. Any hardware-exporter-down silences GPU, thermal and load alerts on the same node. Noise control, by design.

The 12 Dashboards

Provisioned from observability/grafana/dashboards/ into folder MLX Cluster, re-read every 10 seconds, all with a 5-minute auto-refresh. All query VictoriaMetrics — there is no log-content datasource.

Dashboard	Panels	Focus
mlx-cluster	45	The overview — nodes online, req/min, tokens/min, avg tok/s, TTFT p95, plus per-node hardware
mlx-performance	38	KV cache, throughput, hallucination risk, truncation gauge, and the largest token-confidence section
mlx-model-internals	43	Linear-attention internals, weight footprint, ring interconnect counters, shard counts
mlx-node	33	Per-Mac hardware gauges + GPU utilization heatmap + that node’s LLM traffic
mlx-http	33	Request traffic, queue depth, errors, TTFT percentiles, cache hit ratio, entropy and risk heatmaps
mlx-alerts	29	Firing alerts by severity, flapping, supervisor truth, log-tailer health
mlx-model	23	Static architecture from config.json — layers, heads, dims, quantization, context sizing
mlx-call-types	17	Reasoning vs chat, stream vs sync, tool calls and risk broken down by call kind
mlx-gpu-power	15	GPU utilization heatmap, IOKit util, frequency, memory, CPU/GPU/ANE/package watts, thermal pressure
mlx-tools	14	Tool-call usage over 30 d / 7 d / 24 h, calls per minute by tool and type
mlx-logs	12	Metrics about the log pipeline — lines/min, shipped, dropped, queue depth. Not log content.
mlx-sre	18	The two SLOs as gauges, remaining error budget, burn-rate panels

4Alerts & Gaps

46 Alerts In 6 Groups

Thirteen critical, twenty-four warning, nine info — every rule carries team: ml-platform. The two groups in the middle are what make this LLM monitoring rather than generic infrastructure monitoring.

mlx-hardware  15 Rules · 30 s

MLXGPUHighUtilization · MLXGPUMemoryPressure crit · MLXHighPackagePower · MLXPowerSpike · MLXThermalThrottling · MLXCriticalThermalPressure crit · MLXMemoryPressureHigh · MLXLoadHigh · MLXDiskNearlyFull · MLXHardwareExporterDown crit · MLXProxyDown crit · MLXUpstreamDown crit

mlx-serving  8 Rules · 15 s

MLXServerDown crit · MLXModelNotLoaded · MLXServerCrashLoop crit · MLXServerRepeatedCrashes crit · MLXServerReadyProbeFailing · MLXServerMetalFault crit · MLXLogTailerDown · MLXLogTailerNotSending

mlx-inference  7 Rules · 15 s

MLXHighErrorRate · MLXErrorSpike · MLXHighTTFTP95 (>5 s) · MLXSlowGenerationP95 (>30 s) · MLXInferenceStalled crit · MLXInFlightBacklog (>8) · MLXThroughputDrop

mlx-quality  6 Rules · 30 s

Text heuristics. MLXHallucinationRiskCritical (>0.8) crit · MLXHallucinationRiskElevated (>0.5) · MLXHighRepetition · MLXHighHedging · MLXHighUngrounded · MLXHighRefusalRate. All gated on real traffic.

mlx-confidence  5 Rules · 30 s

Sampler logprobs — leads the text heuristics. MLXConfidenceCollapse (margin <0.15) crit — the rule annotation blames “a corrupt or mis-quantised weight load, not hard prompts.” · MLXHighPerplexity · MLXHighTokenEntropy · MLXLowTokenConfidence · MLXConfidenceScoringStopped

mlx-slo  5 Rules · 30 s

Error-budget burn. Availability 99.5 % / 30 d and latency p95 TTFT ≤ 10 s. Fast burn 14.4× critical, slow burn 3× warning, over 1 h and 6 h windows. Plus MLXErrorBudgetExhausted. All gated on traffic > 0 so an idle cluster never pages.

Verified Gaps

These came out of reading the configs against what the code actually emits. Each is a factual mismatch between documentation, config and implementation.

Three alerts can never fire. MLXVictoriaMetricsDown, MLXVmalertDown and MLXAlertmanagerDown match on up{job="victoria-metrics"|"vmalert"|"alertmanager"} — but no scrape job by those names exists in vm-scrape.tmpl.yml. README.md:208-212 documents a self-scrape the template never implements. Self-monitoring is described but absent.

Truncation has a metric and a dashboard panel, but no alert. mlx_finish_reason_total{reason="length"} is charted on mlx-performance.json as “Truncated completions (5m)”. Zero of the 46 rules reference finish_reason. For an agentic or coding workload — where truncation cuts off a half-written thought — this is the most consequential missing alert in the set.

Traces are not persisted by this stack. The traces pipeline exits to the debug exporter only. Opik, reached directly from the proxy, is the system of record. If you query this stack for a trace you will find nothing.

All five images are :latest. victoria-metrics, opentelemetry-collector-contrib, grafana, vmalert, alertmanager. There are also no memory or CPU limits anywhere in the 93-line compose file — for a TSDB holding 30 days of data.

The Grafana datasource has no explicit uid. The provisioning file declares name: VictoriaMetrics but no uid, while all 12 dashboards reference VictoriaMetrics — and two reference lowercase victoriametrics. If panels ever render empty, this is why. The k8s path sidesteps it: apply-dashboards.sh rewrites every uid to mlx-vm at apply time.

One unit trap worth knowing. mlx_hw_mem_pressure is a raw free page count, not bytes and not a 0–1 ratio — despite its help string. MLXMemoryPressureHigh correctly alerts on it as < 20000 pages. Multiply by 4096 for bytes.

Boot Order

cluster/start_server.sh starts telemetry before confirming the server is ready, so hardware metrics flow while the model is still loading.

1. Preflight — verifies both venvs, the prometheus_client import, and passwordless SSH to rank 1. Warns (does not fail) if any of 8081, 8080, 9102, 9104, 9105 or 9106 is still bound.
2. Supervisor → mlx.launch → mlx_lm.server on :8081, ring backend via --hostfile hosts.json, health probe on /v1/models, 5 s interval, exponential backoff 3–30 s.
3. Proxy on :8080 with OTLP and Opik endpoints wired in from cluster.env.
4. Hardware exporter on :9102 — locally for rank 0, and over ssh for rank 1, deliberately in the py3.12 venv to dodge macOS Local-Network TCC blocking.
5. KV agent on :9104, coupled to the server purely by pointing at the same log path the supervisor was told to write.
6. Log tailer on :9106, then wait for /v1/models (180 s TTL) before declaring ready.