New top story on Hacker News: Show HN: RunAnwhere – Faster AI Inference on Apple Silicon
Show HN: RunAnwhere – Faster AI Inference on Apple Silicon
4 by sanchitmonga22 | 0 comments on Hacker News.
Hi HN, we're Sanchit and Shubham (YC W26). We built a fast inference engine for Apple Silicon. LLMs, speech-to-text, text-to-speech – MetalRT beats llama.cpp, Apple's MLX, Ollama, and sherpa-onnx on every modality we tested. Custom Metal shaders, no framework overhead. Also, we've open-sourced RCLI, the fastest end-to-end voice AI pipeline on Apple Silicon. Mic to spoken response, entirely on-device. No cloud, no API keys. To get started: brew tap RunanywhereAI/rcli https://ift.tt/uiIg7hw brew install rcli rcli setup # downloads ~1 GB of models rcli # interactive mode with push-to-talk Or: curl -fsSL https://ift.tt/FuToq0e | bash The numbers (M4 Max, 64 GB, reproducible via `rcli bench`): LLM decode – 1.67x faster than llama.cpp, 1.19x faster than Apple MLX (same model files): - Qwen3-0.6B: 658 tok/s (vs mlx-lm 552, llama.cpp 295) - Qwen3-4B: 186 tok/s (vs mlx-lm 170, llama.cpp 87) - LFM2.5-1.2B: 570 tok/s (vs mlx-lm 509, llama.cpp 372) - Time-to-first-token: 6.6 ms STT – 70 seconds of audio transcribed in *101 ms*. That's 714x real-time. 4.6x faster than mlx-whisper. TTS – 178 ms synthesis. 2.8x faster than mlx-audio and sherpa-onnx. We built this because demoing on-device AI is easy but shipping it is brutal. Voice is the hardest test: you're chaining STT, LLM, and TTS sequentially, and if any stage is slow, the user feels it. Most teams fall back to cloud APIs not because local models are bad, but because local inference infrastructure is. The thing that's hard to solve is latency compounding. In a voice pipeline, you're stacking three models in sequence. If each adds 200ms, you're at 600ms before the user hears a word, and that feels broken. You can't optimize one stage and call it done. Every stage needs to be fast, on one device, with no network round-trip to hide behind. We went straight to Metal. Custom GPU compute shaders, all memory pre-allocated at init (zero allocations during inference), and one unified engine for all three modalities instead of stitching separate runtimes together. MetalRT is the first engine to handle all three modalities natively on Apple Silicon. Full methodology: LLM benchmarks: https://ift.tt/iYZ7F6g... Speech benchmarks: https://ift.tt/PGbtvF0... How: Most inference engines add layers between you and the GPU: graph schedulers, runtime dispatchers, memory managers. MetalRT skips all of it. Custom Metal compute shaders for quantized matmul, attention, and activation - compiled ahead of time, dispatched directly. Voice Pipeline optimizations details: https://ift.tt/YJHLRWF... RAG optimizations: https://ift.tt/lJDe5Hc... RCLI is the open-source voice pipeline (MIT) built on MetalRT: three concurrent threads with lock-free ring buffers, double-buffered TTS, 38 macOS actions by voice, local RAG (~4 ms over 5K+ chunks), 20 hot-swappable models, and a full-screen TUI with per-op latency readouts. Falls back to llama.cpp when MetalRT isn't installed. Source: https://ift.tt/PgLTBM1 (MIT) Demo: https://www.youtube.com/watch?v=eTYwkgNoaKg What would you build if on-device AI were genuinely as fast as cloud?
March 11, 2026 at 12:14AM sanchitmonga22 4 https://ift.tt/sBDRym0 Show HN: RunAnwhere – Faster AI Inference on Apple Silicon 0 Hi HN, we're Sanchit and Shubham (YC W26). We built a fast inference engine for Apple Silicon. LLMs, speech-to-text, text-to-speech – MetalRT beats llama.cpp, Apple's MLX, Ollama, and sherpa-onnx on every modality we tested. Custom Metal shaders, no framework overhead. Also, we've open-sourced RCLI, the fastest end-to-end voice AI pipeline on Apple Silicon. Mic to spoken response, entirely on-device. No cloud, no API keys. To get started: brew tap RunanywhereAI/rcli https://ift.tt/uiIg7hw brew install rcli rcli setup # downloads ~1 GB of models rcli # interactive mode with push-to-talk Or: curl -fsSL https://ift.tt/FuToq0e | bash The numbers (M4 Max, 64 GB, reproducible via `rcli bench`): LLM decode – 1.67x faster than llama.cpp, 1.19x faster than Apple MLX (same model files): - Qwen3-0.6B: 658 tok/s (vs mlx-lm 552, llama.cpp 295) - Qwen3-4B: 186 tok/s (vs mlx-lm 170, llama.cpp 87) - LFM2.5-1.2B: 570 tok/s (vs mlx-lm 509, llama.cpp 372) - Time-to-first-token: 6.6 ms STT – 70 seconds of audio transcribed in *101 ms*. That's 714x real-time. 4.6x faster than mlx-whisper. TTS – 178 ms synthesis. 2.8x faster than mlx-audio and sherpa-onnx. We built this because demoing on-device AI is easy but shipping it is brutal. Voice is the hardest test: you're chaining STT, LLM, and TTS sequentially, and if any stage is slow, the user feels it. Most teams fall back to cloud APIs not because local models are bad, but because local inference infrastructure is. The thing that's hard to solve is latency compounding. In a voice pipeline, you're stacking three models in sequence. If each adds 200ms, you're at 600ms before the user hears a word, and that feels broken. You can't optimize one stage and call it done. Every stage needs to be fast, on one device, with no network round-trip to hide behind. We went straight to Metal. Custom GPU compute shaders, all memory pre-allocated at init (zero allocations during inference), and one unified engine for all three modalities instead of stitching separate runtimes together. MetalRT is the first engine to handle all three modalities natively on Apple Silicon. Full methodology: LLM benchmarks: https://ift.tt/iYZ7F6g... Speech benchmarks: https://ift.tt/PGbtvF0... How: Most inference engines add layers between you and the GPU: graph schedulers, runtime dispatchers, memory managers. MetalRT skips all of it. Custom Metal compute shaders for quantized matmul, attention, and activation - compiled ahead of time, dispatched directly. Voice Pipeline optimizations details: https://ift.tt/YJHLRWF... RAG optimizations: https://ift.tt/lJDe5Hc... RCLI is the open-source voice pipeline (MIT) built on MetalRT: three concurrent threads with lock-free ring buffers, double-buffered TTS, 38 macOS actions by voice, local RAG (~4 ms over 5K+ chunks), 20 hot-swappable models, and a full-screen TUI with per-op latency readouts. Falls back to llama.cpp when MetalRT isn't installed. Source: https://ift.tt/PgLTBM1 (MIT) Demo: https://www.youtube.com/watch?v=eTYwkgNoaKg What would you build if on-device AI were genuinely as fast as cloud? https://ift.tt/w6Xg9Qr
4 by sanchitmonga22 | 0 comments on Hacker News.
Hi HN, we're Sanchit and Shubham (YC W26). We built a fast inference engine for Apple Silicon. LLMs, speech-to-text, text-to-speech – MetalRT beats llama.cpp, Apple's MLX, Ollama, and sherpa-onnx on every modality we tested. Custom Metal shaders, no framework overhead. Also, we've open-sourced RCLI, the fastest end-to-end voice AI pipeline on Apple Silicon. Mic to spoken response, entirely on-device. No cloud, no API keys. To get started: brew tap RunanywhereAI/rcli https://ift.tt/uiIg7hw brew install rcli rcli setup # downloads ~1 GB of models rcli # interactive mode with push-to-talk Or: curl -fsSL https://ift.tt/FuToq0e | bash The numbers (M4 Max, 64 GB, reproducible via `rcli bench`): LLM decode – 1.67x faster than llama.cpp, 1.19x faster than Apple MLX (same model files): - Qwen3-0.6B: 658 tok/s (vs mlx-lm 552, llama.cpp 295) - Qwen3-4B: 186 tok/s (vs mlx-lm 170, llama.cpp 87) - LFM2.5-1.2B: 570 tok/s (vs mlx-lm 509, llama.cpp 372) - Time-to-first-token: 6.6 ms STT – 70 seconds of audio transcribed in *101 ms*. That's 714x real-time. 4.6x faster than mlx-whisper. TTS – 178 ms synthesis. 2.8x faster than mlx-audio and sherpa-onnx. We built this because demoing on-device AI is easy but shipping it is brutal. Voice is the hardest test: you're chaining STT, LLM, and TTS sequentially, and if any stage is slow, the user feels it. Most teams fall back to cloud APIs not because local models are bad, but because local inference infrastructure is. The thing that's hard to solve is latency compounding. In a voice pipeline, you're stacking three models in sequence. If each adds 200ms, you're at 600ms before the user hears a word, and that feels broken. You can't optimize one stage and call it done. Every stage needs to be fast, on one device, with no network round-trip to hide behind. We went straight to Metal. Custom GPU compute shaders, all memory pre-allocated at init (zero allocations during inference), and one unified engine for all three modalities instead of stitching separate runtimes together. MetalRT is the first engine to handle all three modalities natively on Apple Silicon. Full methodology: LLM benchmarks: https://ift.tt/iYZ7F6g... Speech benchmarks: https://ift.tt/PGbtvF0... How: Most inference engines add layers between you and the GPU: graph schedulers, runtime dispatchers, memory managers. MetalRT skips all of it. Custom Metal compute shaders for quantized matmul, attention, and activation - compiled ahead of time, dispatched directly. Voice Pipeline optimizations details: https://ift.tt/YJHLRWF... RAG optimizations: https://ift.tt/lJDe5Hc... RCLI is the open-source voice pipeline (MIT) built on MetalRT: three concurrent threads with lock-free ring buffers, double-buffered TTS, 38 macOS actions by voice, local RAG (~4 ms over 5K+ chunks), 20 hot-swappable models, and a full-screen TUI with per-op latency readouts. Falls back to llama.cpp when MetalRT isn't installed. Source: https://ift.tt/PgLTBM1 (MIT) Demo: https://www.youtube.com/watch?v=eTYwkgNoaKg What would you build if on-device AI were genuinely as fast as cloud?
March 11, 2026 at 12:14AM sanchitmonga22 4 https://ift.tt/sBDRym0 Show HN: RunAnwhere – Faster AI Inference on Apple Silicon 0 Hi HN, we're Sanchit and Shubham (YC W26). We built a fast inference engine for Apple Silicon. LLMs, speech-to-text, text-to-speech – MetalRT beats llama.cpp, Apple's MLX, Ollama, and sherpa-onnx on every modality we tested. Custom Metal shaders, no framework overhead. Also, we've open-sourced RCLI, the fastest end-to-end voice AI pipeline on Apple Silicon. Mic to spoken response, entirely on-device. No cloud, no API keys. To get started: brew tap RunanywhereAI/rcli https://ift.tt/uiIg7hw brew install rcli rcli setup # downloads ~1 GB of models rcli # interactive mode with push-to-talk Or: curl -fsSL https://ift.tt/FuToq0e | bash The numbers (M4 Max, 64 GB, reproducible via `rcli bench`): LLM decode – 1.67x faster than llama.cpp, 1.19x faster than Apple MLX (same model files): - Qwen3-0.6B: 658 tok/s (vs mlx-lm 552, llama.cpp 295) - Qwen3-4B: 186 tok/s (vs mlx-lm 170, llama.cpp 87) - LFM2.5-1.2B: 570 tok/s (vs mlx-lm 509, llama.cpp 372) - Time-to-first-token: 6.6 ms STT – 70 seconds of audio transcribed in *101 ms*. That's 714x real-time. 4.6x faster than mlx-whisper. TTS – 178 ms synthesis. 2.8x faster than mlx-audio and sherpa-onnx. We built this because demoing on-device AI is easy but shipping it is brutal. Voice is the hardest test: you're chaining STT, LLM, and TTS sequentially, and if any stage is slow, the user feels it. Most teams fall back to cloud APIs not because local models are bad, but because local inference infrastructure is. The thing that's hard to solve is latency compounding. In a voice pipeline, you're stacking three models in sequence. If each adds 200ms, you're at 600ms before the user hears a word, and that feels broken. You can't optimize one stage and call it done. Every stage needs to be fast, on one device, with no network round-trip to hide behind. We went straight to Metal. Custom GPU compute shaders, all memory pre-allocated at init (zero allocations during inference), and one unified engine for all three modalities instead of stitching separate runtimes together. MetalRT is the first engine to handle all three modalities natively on Apple Silicon. Full methodology: LLM benchmarks: https://ift.tt/iYZ7F6g... Speech benchmarks: https://ift.tt/PGbtvF0... How: Most inference engines add layers between you and the GPU: graph schedulers, runtime dispatchers, memory managers. MetalRT skips all of it. Custom Metal compute shaders for quantized matmul, attention, and activation - compiled ahead of time, dispatched directly. Voice Pipeline optimizations details: https://ift.tt/YJHLRWF... RAG optimizations: https://ift.tt/lJDe5Hc... RCLI is the open-source voice pipeline (MIT) built on MetalRT: three concurrent threads with lock-free ring buffers, double-buffered TTS, 38 macOS actions by voice, local RAG (~4 ms over 5K+ chunks), 20 hot-swappable models, and a full-screen TUI with per-op latency readouts. Falls back to llama.cpp when MetalRT isn't installed. Source: https://ift.tt/PgLTBM1 (MIT) Demo: https://www.youtube.com/watch?v=eTYwkgNoaKg What would you build if on-device AI were genuinely as fast as cloud? https://ift.tt/w6Xg9Qr
Nhận xét
Đăng nhận xét