Local LLM Architecture

Homelab Proxmox Cluster • High-Performance Compute Node

Physical Infrastructure

  • Intel Core i5 (14th Gen) High-performance P-core/E-core architecture
  • AMD Radeon RX 6750 XT Navi 22 • 12GB VRAM • ROCm Accelerated
  • 32GB High-Speed RAM Low-latency memory for localized model inference

Virtualized Environment

OS: Linux (Containerized)
  • Provisioned Cores 8 Dedicated
  • Provisioned RAM 16 GB
  • Storage NVMe Backend
  • Hardware Passthrough PCI-e Compute Passthrough (Direct) Hardware-Accelerated Render Engine

Active Inference Stack

Engine Ollama Running on Port 11434
Compute API ROCm Stack AMD Open Ecosystem
Host OS Proxmox VE Debian 12 Bookworm Based

Current Active Models

Qwen 2.5 Coder 14B Quantization: Q4_K_M
8.4 GB
Qwen 2.5 Coder 14B Aider Fine-tuned for agentic workflows
8.4 GB