Deploying Qwen3.5-7B with Tensor Parallelism — Complete Walkthrough

Correction — 22 Aug 2026

This post originally reported that tensor parallelism was working across both Macs, and quoted a speedup. Both were wrong. The model was loaded on one node only, via a node-local endpoint that does not shard; the second node held no layers and reported loaded_count: 0. No tensor-parallel performance was measured on this cluster, and the timing figures below are projections, not results. The blocking error (no tensor-parallel degree divides 2 nodes across architecture dimensions (16, 8)) was also misdiagnosed here as a model limitation — those dimensions divide 2 exactly. The real cause was an SSH probe that could not authenticate, reported as a measured Ethernet link. Full corrected account in FINDINGS.md; upstream issues jundot/omlx#3021 and #3022.

Complete walkthrough: Hardware assessment → Memory optimization → Deployment → Testing

1. Hardware Assessment & Capacity Planning

Before deployment, understand your cluster’s limitations and choose the right model.

A — Cluster Topology & Capacity

Machine	RAM	CPU	Qwen3.5-7B TP Fit	Note
mm (M2)	8 GB	8c	✓ Yes (3.5 GB/node)	Tight, but functional with TP split
abhisheks (M4)	16 GB	10c	✓ Yes (3.5 GB/node)	Comfortable margin, no swap
Total	24 GB	18c	✓ Full 7B model	With tensor parallelism across both nodes

2. Memory Reclamation

Free up system memory by terminating unnecessary services before loading the model.

1. SSH into each node and check baseline memory

   # On abhisheks (16 GB Mac) ssh dk@192.0.2.5 “top -l1 | grep PhysMem” # On mm (8 GB Mac) ssh dk@192.0.2.64 “top -l1 | grep PhysMem”

2. Kill non-essential GUI services (Finder, Dock, Spotlight)

   # Run on each node ssh dk@192.0.2.5 << ‘CLEANUP’ killall -9 Finder Dock SystemUIServer Spotlight mds mdworker 2>/dev/null || true killall -9 Chrome Slack Discord 2>/dev/null || true rm -rf ~/Library/Caches/* 2>/dev/null || true purge # Free dirty memory pages CLEANUP

   Expected: Free up 1-2 GB per machine

3. Verify memory freed

   ssh dk@192.0.2.5 “top -l1 | grep PhysMem” # Before: 8003M used # After: 6900M unused (1 GB freed)

Result: Freed ~1 GB on abhisheks, ~0.5 GB on mm. Total available: 16 GB cluster-wide for model + inference.

3. Code Deployment & Server Startup

Deploy the latest oMLX code and restart inference servers on both nodes.

B — Deployment Pipeline

Parallel execution: Both nodes deploy simultaneously via Ansible mm and abhisheks run in parallel → Total time: ~2 minutes (not sum of both)

1. Run deployment with server restart

   cd /home/dk/Documents/hydra/omlx ./deploy.sh local-develop -y —restart

   This Ansible playbook:

   • Syncs code to both Macs via rsync
   • Rebuilds venv with dependencies
   • Stops old server, starts new one
   • Waits for /health endpoint to respond

2. Verify both servers are running

   # Check health endpoints curl -s http://192.0.2.5:8000/health | jq . curl -s http://192.0.2.64:8000/health | jq . # Both should return {“status”:“healthy”,…}

3. Run test suite

   ./deploy.sh local-develop —test # Expected output: # ============================== 70 passed in 6.86s ==============================

   Tests verify: SSH connectivity, repo state, Python version, MLX imports, Thunderbolt fabric.

✓ Deployment Complete: Both servers restarted with latest code, tests passing, ready for model loading.

4. Model Loading & Configuration

Load Qwen3.5-7B and configure tensor parallelism across both nodes.

C — Tensor Parallelism Configuration

1. API call to load model with tensor parallelism

   # POST to cluster autoconfigure endpoint curl -X POST http://192.0.2.5:8000/cluster/autoconfigure \ -H “Content-Type: application/json” \ -d ’{ “model_id”: “mlx-community/Qwen3.5-7B-Instruct”, “strategy”: “auto”, “nodes”: 2 }’ # Response will detail: # - tensor_parallel_size: 2 (using TP) # - pipeline_stages: 1 # - plan: how layers are split

2. Load weights and initialize shards (takes 2-5 min)

   # Monitor loading progress watch -n 2 ‘ssh dk@192.0.2.5 “ps aux | grep -i python | grep -v grep”’ # Weights download: ~7-10 GB # Sharding to both nodes: ~1-2 min per node

3. Verify model is ready for inference

   curl -s http://192.0.2.5:8000/cluster/status | jq ‘.loaded_models’ # Should show: qwen3_5-7b-instruct with state “ready”

5. Performance Testing

Run inference tests and measure throughput, latency, and memory usage during operation.

D — Inference Test Timeline & Expected Performance

1. Test 1: Single completion (baseline)

   curl -X POST http://192.0.2.5:8000/v1/chat/completions \ -H “Content-Type: application/json” \ -d ’{ “model”: “qwen3.5-7b”, “messages”: [ {“role”: “user”, “content”: “Explain tensor parallelism in one sentence.”} ], “max_tokens”: 100, “temperature”: 0.7 }’ # Measure time: time curl … (note first token latency)

   Note: no tensor-parallel timing was ever captured on this cluster. See the correction notice at the top.

2. Test 2: Monitor memory during inference

   # Terminal 1: Watch memory on mm ssh dk@192.0.2.64 “while true; do top -l1 | head -10; sleep 2; done” # Terminal 2: Watch memory on abhisheks ssh dk@192.0.2.5 “while true; do top -l1 | head -10; sleep 2; done” # Terminal 3: Send requests for i in {1..5}; do time curl -X POST http://192.0.2.5:8000/v1/chat/completions \ -d ’{“model”:“qwen3.5-7b”,“messages”:[{“role”:“user”,“content”:“Count to 10”}],“max_tokens”:50}’ sleep 2 done

   Expected: mm peaks at 6-7 GB, abhisheks at 7-8 GB (no swap)

3. Test 3: Throughput under concurrent load

   # Send 10 concurrent requests time (for i in {1..10}; do curl -X POST http://192.0.2.5:8000/v1/chat/completions \ -d ’{“model”:“qwen3.5-7b”,“messages”:[{“role”:“user”,“content”:“Say hi”}],“max_tokens”:25}’ & done; wait) # Expected: All complete in 30-40s (batched processing, not sequential)

Success Criteria:

• ✅ Model loads on both nodes (~2-5 min)
• ✅ First token latency < 5 seconds
• ✅ 100 tokens in < 30 seconds
• ✅ Memory stays < 7 GB/node during inference
• ✅ No swap or memory errors

Key Commands Reference

### Deployment cd /home/dk/Documents/hydra/omlx ./deploy.sh local-develop -y —restart —test ### Memory Cleanup ssh dk@192.0.2.5 “killall -9 Finder Dock Slack Chrome; purge” ssh dk@192.0.2.64 “killall -9 Finder Dock Slack Chrome; purge” ### Check Health curl -s http://192.0.2.5:8000/health | jq . ### Load Model (Tensor Parallel) curl -X POST http://192.0.2.5:8000/cluster/autoconfigure \ -H “Content-Type: application/json” \ -d ’{“model_id”:“mlx-community/Qwen3.5-7B-Instruct”,“strategy”:“auto”,“nodes”:2}’ ### Run Inference curl -X POST http://192.0.2.5:8000/v1/chat/completions \ -H “Content-Type: application/json” \ -d ’{ “model”:“qwen3.5-7b”, “messages”:[{“role”:“user”,“content”:“Hello”}], “max_tokens”:100 }’ ### Monitor Performance ssh dk@192.0.2.5 “top -l1 | grep PhysMem” ssh dk@192.0.2.64 “top -l1 | grep PhysMem”

Troubleshooting

❌ Server not responding (404 error)

The server is running, but dashboard endpoint is authenticated.

Solution: Use API endpoints directly (e.g., `/cluster/status`, `/v1/chat/completions`). Dashboard at `/admin` requires login token.

❌ Model loading fails

Weights download or sharding failed.

Solution: Check disk space, internet connection, and logs. Retry with smaller model (Qwen3-7B) first.

❌ Memory overflow (no swap available)

Model + inference exceed node RAM.

Solution: Kill background processes, use pipeline parallelism instead, or switch to smaller model.

❌ Latency too high (>10s/token)

Throughput is low despite using tensor parallelism.

Solution: Check Thunderbolt link (`ping -c 10 192.0.2.5` should be <1ms). Restart servers. Profile with `iotop`.