Last year I wrote about hacking the RK3588 NPU to run a vision transformer. The chip’s own SDK refused to run the model, so I reverse-engineered the hardware limits and rewrote the graph by hand to make the closed toolchain cooperate. It worked. It was also fragile, because the toolchain stayed a black box.

This year I went further. Instead of fighting the black box, I opened it. I reverse-engineered the register commands the NPU actually executes, then built a compiler and runtime that talk to the hardware directly. No closed SDK. No vendor runtime. The result is OpenNPU, an MIT-licensed toolkit that runs GPT-2 and SigLIP from PyTorch, ONNX, and JAX, and that you can extend with your own operators.

This post is the story of that year. For the technical details, the wire format, the compiler pipeline, and the measured numbers, see the companion report: OpenNPU v1.0: an open compiler and runtime for the RK3588 NPU.

The year's arc Last year we worked around the closed toolchain. This year we opened it.

Last year: the vision encoder hack

The RK3588 is a $100 chip with a 6 TOPS NPU. TOPS means trillions of operations per second, and 6 TOPS is a lot for the price. The catch is how you reach it. Rockchip ships the NPU behind two closed pieces of software: a compiler that turns a model into a binary file, and a runtime that loads that file and drives the hardware.

I wanted to run the SigLIP vision encoder from SmolVLM. The closed compiler refused. It threw an undocumented error, REGTASK Overflow (0xe010), and gave up. The model’s attention matrices were too big for the NPU’s small on-chip scratch memory, and the compiler had no way to handle them.

So I did the surgery myself. I sliced the attention into small tiles that fit the scratch memory, injected a tiny dummy operation to stop the compiler from fusing my tiles back into one giant block, and scaled the activations so the model’s wide value range survived quantization. It worked. The encoder ran 15x faster than the CPU, and the output matched the reference.

That post ended there. The model ran. But the approach had a ceiling.

The itch: a black box you cannot extend

The gray-box fix worked for one model, but it was hand surgery. Every new model needed the same treatment: find the constraint, tile around it, trick the compiler. The toolchain itself stayed opaque. I could not add an operator it did not know. I could not change how it scheduled work. I could not ship the result, because the whole thing depended on a closed compiler and a closed runtime.

Two ways to reach the NPU The closed path hides the hardware behind two opaque layers. The open path talks to it directly.

The deeper problem was that reverse engineering the hardware limits was not the same as understanding the hardware. I knew the NPU had a small scratch memory. I did not know what bytes it actually executed. To build something reusable, I needed the format underneath.

Opening the box

The NPU is a computer. Like any computer, it runs instructions. Rockchip does not document those instructions, but the runtime has to issue them, and the runtime is a file on disk. I captured the bytes the runtime sent to the NPU and decoded the format.

The format turned out to be simple: the NPU is programmed by writing eight-byte register-command entries, and the runtime submits a chain of command blocks. Once the format was known, the closed runtime became optional. I could write the bytes myself and submit them through the public DRM ioctl interface. The report walks the format entry by entry.

Building the compiler

With the format in hand, I built a compiler. It takes a model graph, lowers it through a small IR, and emits the register commands. The whole thing is pure Python, a few hundred lines, with no dependency on LLVM or MLIR.

The compiler has one important trick. The NPU has no general matrix multiply instruction, so the compiler implements Y = X @ W as a one-by-one convolution, which the hardware calls a CNA descriptor. That single trick is what lets any transformer run: attention and the MLP layers are just chains of matrix multiplies. The report covers the descriptor and the pipeline in detail.

The framework: PyTorch, ONNX, and JAX

The compiler alone is not a toolkit. A toolkit is something you can call from the framework you already use. So OpenNPU exposes the same path through three front ends.

PyTorch gets the NPU as a device. You build a model with torch.nn, move it to the NPU, and run it. ONNX gets a runner that walks an exported graph and dispatches each node to the NPU or the CPU. JAX gets a real PJRT plugin, the same interface JAX uses for GPUs and TPUs, so jax.jit dispatches to the NPU and jax.devices() lists it as npu:0.

The JAX plugin is the most involved. It is pinned to a specific PJRT API version, because JAX reads the plugin’s function table at the offsets of the exact version it was compiled against. It parses the model bytecode JAX sends it, recognizes a small set of operations, and runs them on the NPU. The fast matmul path is exposed as a custom call that skips the parsing entirely.

Extending it

The point of a framework is that you can add to it. OpenNPU documents two routes for a new operator.

For a matrix-multiply-shaped operator, use the CNA descriptor path. The descriptor is parameterized, so a new shape or a new fused pattern is a configuration change, not a rewrite. For an elementwise or activation operator, generate a C template and let the plugin dispatch to it. Either way, the operator becomes callable from PyTorch and JAX once it is in the toolkit.

That is the difference from a one-off hack. Last year, adding an operator meant redoing the surgery. This year, it means adding a case to a table.

The results

Throughput on the RK3588 NPU GPT-2 decodes at 36 tok/s with KV caching. SigLIP runs at 1.1 img/s, or 2.7 img/s across three cores.

The GPT-2 decoder output matches PyTorch token for token. The SigLIP encoder matches the HuggingFace reference to cosine similarity 0.9999. Forty of the forty-one op and precision combinations pass on board. The report has the full comparison table and the honest hardware limits.

What it means

Reverse engineering the RK3588 NPU is not new. mtx512/rk3588-npu did it first, in early 2024, on the older 5.10 kernel, and the write-up is worth reading. Their approach was different: they derived the CNA and DPU register descriptors from the chip’s technical reference manual and ran matmul on the NPU’s internal SRAM. The register and submit formats changed in the newer 6.1 driver, so the descriptor here was re-derived on this kernel rather than ported; the older code does not run correctly on 6.1. The CNA descriptor matmul is one piece of that work, not the whole of it. The larger part was capturing the register-command templates and decoding the wire format, which is what the compiler and the forty op and precision combinations rest on. The contribution is the framework built on top of that: a compiler and runtime that work from PyTorch, ONNX, and JAX, that cover forty op and precision combinations, and that you can extend with your own operators.

The goal is a toolkit that makes the NPU easier to work on, not a claim about who did the reverse engineering first. The code is MIT licensed at github.com/poad42/opennpu_rk3588. The CNA descriptor matmul builds on prior work by mtx512/rk3588-npu, which is credited in the repository. If you have an RK3588 board, the examples are a good place to start.