DEV Community

Abhishek Khanra
Abhishek Khanra

Posted on Edited on AI-assisted

I Fixed Microsoft's BitNet C++ Kernels: How to Run 1-Bit LLMs on Pure CPU Without Compiling

#cpp #ai #opensource #machinelearning

https://github.com/IshekKhal/bitrun

I built BitRun because I wanted to run 1-bit language models on my laptop without melting my battery, burning up a GPU, or wrestling with broken build systems.

Right now, BitRun runs ternary models at 98 to 127 tokens per second on an Apple Silicon CPU and 40 to 45 tokens per second on an Intel PC CPU. It uses around 1.3 GB of RAM and leaves the GPU at 0%.

Getting to those numbers was not easy.

When Microsoft Research published the BitNet b1.58 paper, the concept sounded revolutionary: train language models from scratch using ternary weights {-1, 0, +1} so you can completely eliminate floating-point matrix multiplications. During inference, your CPU computes tensor operations using simple integer additions.

That promised 70% to 80% lower energy draw and extreme CPU speeds on ordinary laptops.

Naturally, I wanted to run it. I cloned Microsoft's official research repository (bitnet.cpp), opened my terminal, and tried to build it.

That was when everything caught on fire.

When Code Compiles But the Output Is Garbage

Getting Microsoft's reference code to run properly on modern hardware was a nightmare:

  • On Windows, CMake threw fatal errors if you used standard Visual Studio MSVC.
  • On Apple Silicon (ARM64), when Microsoft released their official 2.4B model checkpoint (bitnet-b1.58-2B-4T-gguf), it went completely off the rails on ARM chips. It either crashed on multi-turn conversations or got stuck in an infinite loop spitting out: 
  @ @ @ @ @ @ @ @ @ @ @ @ @ @ @
Enter fullscreen mode Exit fullscreen mode
  • The setup required a labyrinth of dependencies: recursive submodules, specific Clang toolchains, Python virtual environments, or pulling an 8 GB Docker image just to test a 1 GB model.

I wanted to run a model on my CPU, not maintain a small operating system.

So I pulled down the C++ source, fired up the debugger, and started stepping through the math kernels myself.

The Detective Work: 5 Bugs in Microsoft's Kernels

Digging through the low-level math kernels revealed five separate bugs causing the corruption:

1. The Disappearing Batch Rows (src/ggml-bitnet-mad.cpp)

In the Multiply-Add (MAD) kernels, the output matrix was indexed as s[row] instead of accounting for the batch stride (bs). During batch operations, every new row wrote directly to index row, silently overwriting preceding rows in memory. Standardized all output writes across AVX2 and ARM NEON kernels to s[(row + rb) * bs].

2. The 64 vs 128 Pointer Shift (ggml-cpu-i2s.c)

In ggml-cpu-i2s.c, upstream defined block size as #define QK_I2_S 64 when compiling for ARM NEON. But the official GGUF ternary quantization format standardizes on 128 elements per block (QK_I2_S 128). That single mismatch shifted pointer arithmetic across all weight blocks by 50%. The ARM DotProd SIMD adders were processing half-shifted memory garbage, which explained the repeating @ tokens. Unified #define QK_I2_S 128 across all CPU paths.

3. Wrong Activation Function (src/models/bitnet.cpp)

In src/models/bitnet.cpp, the model graph initialized the feed-forward network with SiLU. But Microsoft trained the 2.4B model with squared ReLU (RELU_SQR). SiLU has a completely different activation curve, scrambling outputs at every layer. Updated the model definition in C++ to use LLM_FFN_RELU_SQR.

4. The 32-Token Segfault (ggml-blas.cpp)

During multi-turn chat, once context passed 32 tokens, the process crashed with EXC_BAD_ACCESS. The BLAS backend was claiming ternary matrix operations and jumping past buffer boundaries with unaligned offsets. Excluded ternary tensors from BLAS dispatch so math stays on the verified CPU SIMD kernels, and corrected tensor row size calculations.

5. The Windows MSVC Lockout (src/CMakeLists.txt)

In src/CMakeLists.txt, CMake checked for Clang or GCC and aborted configuration with a FATAL_ERROR if neither was detected. The underlying C++17 kernels actually compile cleanly with MSVC once you remove the check and enable standalone CLI tools targets.

I submitted the fixes upstream to Microsoft in two pull requests:

  • microsoft/BitNet#642: Fixes ARM64 batch stride write-backs, block size alignment, squared ReLU activation, and BLAS crashes.
  • microsoft/BitNet#644: Enables native MSVC compiler support and CLI build targets on Windows.

Both PRs are currently open and passing CI on Microsoft's repository.

Enter BitRun: 1-Bit LLMs With Zero Compiling

Fixing the C++ math kernels made the models speak articulate English, but nobody wants to fight C++ compilers and submodules just to test a local model.

To make running these models zero-friction, I built BitRun: an open-source, self-contained standalone runner with zero external dependencies.

Think of it as Ollama, but purpose-built for 1-bit ternary models on CPU.

Under the hood, BitRun pairs a Go supervisor with a native statically compiled C++ engine:

  • Supervisor Layer (Go): Handles hardware topology detection, model downloading from Hugging Face, Modelfile persona parsing, and an OpenAI-compatible REST server.
  • Engine Tier (Native C++): Statically compiled binary utilizing ARM NEON with dot-product (ARM_FEATURE_DOTPROD) on Apple Silicon and AVX2 on x86, with zero CGO dependencies for total crash isolation. 
+-------------------------------------------------------------+
|                     BitRun CLI (Go)                         |
|   pull  |  run  |  serve  |  list  |  create -f Modelfile   |
+-------------------------------------------------------------+
                              |
       (Process boundary / pipes / HTTP supervisor)
                              |
+-------------------------------------------------------------+
|              Native Inference Engine (C++)                  |
|   ARM NEON DotProd (Apple)  |  AVX2 / AVX-512 SIMD (x86)    |
|   Ternary {-1, 0, +1} Integer Add Kernels (BitNet b1.58)    |
+-------------------------------------------------------------+
Enter fullscreen mode Exit fullscreen mode

Real CPU Speeds (Pure CPU, 0% GPU)

Here is what inference speeds look like on standard hardware with GPU offloading completely disabled (-ngl 0):

1. Apple Silicon (M4 Pro CPU, 12 Cores, ARM NEON DotProd)

Model Parameters Format Disk Size Prompt Speed Generation Speed Memory (RAM)
Falcon3 1B Instruct 1B i2_s Ternary ~1.30 GB 137–142 t/s 98–127 t/s ~1.9 GB
BitNet b1.58 2.4B 2.4B i2_s Ternary ~1.13 GB 60.1 t/s 39.8 t/s ~1.8 GB
BitNet 70M 70M Q8_0 ~79 MB 8,758 t/s 1,355 t/s < 150 MB

2. Windows x86_64 (Intel Core i5-12600K, 16 Threads, AVX2)

Model Parameters Format Disk Size Prompt Speed Generation Speed Memory (RAM)
Falcon3 1B Instruct 1B i2_s Ternary ~1.30 GB 209.6 t/s 39–45 t/s ~1.9 GB
BitNet 70M 70M Q8_0 ~79 MB 2,419 t/s 492 t/s < 150 MB

3. Linux x86_64 (Ubuntu / WSL2, 4 Cores, AVX2)

Model Parameters Format Disk Size Prompt Speed Generation Speed Memory (RAM)
Falcon3 1B Instruct 1B i2_s Ternary ~1.30 GB ~130 t/s 40–45 t/s ~1.9 GB
BitNet 70M 70M Q8_0 ~79 MB ~2,100 t/s ~450 t/s < 150 MB

How to Try It in 60 Seconds

You do not need CMake, Python, or Docker installed.

1. Install BitRun

On macOS (Apple Silicon) or Linux:

curl -fsSL https://raw.githubusercontent.com/IshekKhal/bitrun/main/install.sh | sh
Enter fullscreen mode Exit fullscreen mode

On Windows (PowerShell):

irm https://raw.githubusercontent.com/IshekKhal/bitrun/main/install.ps1 | iex
Enter fullscreen mode Exit fullscreen mode

2. Pull and Chat 

# Pull model weights from Hugging Face
bitrun pull falcon3-1b

# Start chatting on pure CPU
bitrun run falcon3-1b

# Run a one-shot query directly from terminal
bitrun run falcon3-1b "Explain RAII in C++"
Enter fullscreen mode Exit fullscreen mode

3. Spin Up an OpenAI-Compatible API

You can also run BitRun as a local background server on port 11435:

bitrun serve falcon3-1b
Enter fullscreen mode Exit fullscreen mode

Point Cursor, Continue.dev, Open WebUI, or the standard OpenAI Python client directly to http://localhost:11435/v1:

curl -s http://127.0.0.1:11435/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "falcon3-1b",
    "messages": [
      {"role": "user", "content": "What is the capital of France?"}
    ],
    "temperature": 0.7
  }'
Enter fullscreen mode Exit fullscreen mode

4. Custom Personas via Modelfiles

You can define custom system prompts using Modelfiles:

# Modelfile
FROM falcon3-1b
SYSTEM You are an expert systems engineer. Write concise, idiomatic C code.
PARAMETER temperature 0.2
Enter fullscreen mode Exit fullscreen mode

Build and run your persona:

bitrun create engineer -f Modelfile
bitrun run engineer
Enter fullscreen mode Exit fullscreen mode

Where 1-Bit AI Actually Stands Today

Ternary weights require Quantization-Aware Training from scratch; you cannot take an existing 70B FP16 model and post-quantize it down to 1.58 bits without wrecking its outputs.

Because pre-training is compute-intensive, the public catalog of ternary checkpoints is currently limited to models under 3 billion parameters (falcon3-1b, bitnet-2b, falcon3-3b, bitnet-70m). These will not replace Claude or GPT-4 for complex coding.

However, we are in the infrastructure phase of ternary models. Having a fast, zero-dependency C++ runtime and model manager ready today means:

  1. Edge and IoT Applications: You can run local classification, entity extraction, and Home Assistant automations on low-power Intel N100 mini PCs or Raspberry Pi boards with near-zero wattage.
  2. Battery Efficiency: You can run background agents on a laptop without draining battery life or spinning up cooling fans.
  3. Prepared for 7B+ Models: When open research labs release 7B and 14B ternary weights, the runtime to execute them at 40+ tokens/sec on ordinary CPUs will already be waiting.

Open Source Links

All code is open source under the MIT license:

Top comments (0)