CSCI 7000-011 — Current Topics in CS: Transformers for Robotics Instructor: Nikolaus Correll · CU Boulder · Spring 2026

Paper: “No Plan but Everything Under Control” — Mengers & Brock, ICRA 2025
My implementation: github.com/poad42/no_plan_everything_control (pure PyTorch, runnable without simulation)

The Problem Every Robotics Student Hits

You’re building a robot that needs to open a drawer. Sounds simple, right? But think about everything that has to happen:

1. The robot doesn’t know where the drawer handle is → it needs to look around first
2. Once it sees the handle, it needs to reach for it
3. It needs to grasp the handle
4. Then pull the drawer open

Traditional approaches say: write a planner. Define each sub-goal. Build a state machine that transitions between them. Handle failures at each stage. If something goes wrong mid-execution… re-plan.

This paper throws all of that away. Their claim: if you set up the math right, gradient descent alone will figure out the correct ordering of sub-goals. No planner. No state machine. No learned policy. Just calculus.

The Big Idea in One Sentence

Wire your sensors, actuators, and goal into a single differentiable graph. Take the gradient. Follow the steepest one. Repeat.

That’s it. The rest of the paper is explaining why this works and how to set up the graph correctly.

The Algorithm at a Glance

The entire algorithm is a loop:

1. Components update their state estimates from sensor data (Eq. 1)
2. Interconnections compute soft gates — p_visible, p_grasped, etc. (Eq. 2)
3. Goal cost is differentiated through all paths in the graph (Eq. 3)
4. The steepest gradient determines the next action (Eq. 4)
5. The action changes the world → new sensor data → repeat

There is no planner, no policy, no search. The feedback arrow closes the loop. Let’s look at each piece.

Building Block 1: Components (Eq. 1)

Think of a “component” as a little module that tracks one thing about the world. The robot’s end-effector position. The estimated location of the drawer handle. Whether the gripper is open or closed.

Each component updates its estimate at every timestep:

x_t = f(x_{t-1}; c_1, c_2, ..., c_N)

“My new estimate depends on my old estimate plus whatever information is flowing in from other components.”

Here’s what this looks like in practice — an EKF tracking the drawer handle position (components.py):

class EKFComponent(BaseComponent):
    """Extended Kalman Filter — tracks one quantity with uncertainty."""

    def update(self, priors):
        # Predict step
        x_pred, F = self._process_model(self._state, priors)
        Sigma_pred = F @ self._Sigma @ F.T + self._Q

        # Update step (if we have a measurement)
        if "measurement" in priors:
            z = priors["measurement"]
            z_pred, H = self._measurement_model(x_pred)
            S = H @ Sigma_pred @ H.T + self._R
            K = Sigma_pred @ H.T @ torch.linalg.solve(S, torch.eye(...))
            self._state = x_pred + K @ (z - z_pred)
            self._Sigma = (I - K @ H) @ Sigma_pred
        else:
            self._state = x_pred
            self._Sigma = Sigma_pred

Everything is PyTorch, so gradients flow through torch.autograd. Not just an estimator — a differentiable estimator.

Building Block 2: Active Interconnections (Eq. 2)

Components share information through differentiable gates that turn on or off depending on the current state:

c(x_1, x_2, ..., x_M)    — coupling changes based on state

Consider a visibility gate. Whether the wrist camera can see the drawer handle depends on where the end-effector is pointing. So p_visible is a soft sigmoid (interconnections.py):

class VisibilityGate(BaseInterconnection):
    def forward(self, ee_pose, object_pose):
        direction = object_pose[:3] - ee_pose[:3]
        cos_angle = F.cosine_similarity(direction, camera_forward, dim=0)
        return torch.sigmoid(self._sharpness * (cos_angle - self._threshold))

And p_grasped uses force-torque sensor readings:

class GraspGate(BaseInterconnection):
    def forward(self, distance, grip_force, hand_closed):
        p_close = torch.sigmoid(-self._dist_sharpness * (distance - self._dist_thresh))
        p_force = torch.sigmoid(self._force_sharpness * (grip_force - self._force_thresh))
        p_hand  = torch.sigmoid(self._hand_sharpness * (hand_closed - 0.5))
        return p_close * p_force * p_hand

This is the key insight. When p_visible = 0, the gradient path through vision is blocked. The algorithm naturally concludes: “I can’t optimize the grasp yet. The steepest gradient tells me to move the camera first.” Subgoal ordering falls out of the math — nobody programmed it.

Building Block 3: Steepest Gradient Action Selection (Eqs. 3–4)

The goal is a differentiable scalar cost. For opening a drawer 20 cm:

g(q) = (q − 0.20)²

The gradient propagates backward through all interconnections and components via the chain rule. Because different gates are open or closed, you get multiple gradient paths — each representing a different possible sub-goal.

The rule: pick the steepest one (gradient_descent.py):

# Compute all gradient paths
gradients = [path.gradient(action) for path in self._paths]

# Select steepest (Eq. 4)
norms = torch.stack([g.norm() for g in gradients])
best_idx = int(torch.argmax(norms).item())
grad_star = gradients[best_idx]

# Gradient descent (Eq. 3)
action_new = action - k * grad_star

That’s the entire control loop. No planner. No policy network.

Why It Works: The Locked Door Example

A point mass needs to reach (10, 0), but a wall at x=5 is locked. A button at (2, 4) opens it. The complete AICON implementation (demo_2d_locked_door.py):

class PointMassAICON:
    def step(self, lr=0.15):
        pos = self.agent_pos
        goal_dist = torch.norm(pos - self.goal_pos)
        dist_to_button = torch.norm(pos - self.button_pos)

        p_button_soft = torch.exp(-0.25 * (dist_to_button ** 2))
        door_current = door_prev + p_button * (1.0 - door_prev)

        factor = 1.0 + 10.0 * (1.0 - door_current)
        cost = goal_dist * factor + barrier * (1.0 - door_current)

        cost.backward()  # PyTorch does the rest

When the door is closed (door_current ≈ 0), factor = 11× — making the goal gradient weak relative to the button gradient. The agent:

1. Moves toward the button (steepest gradient when door is closed)
2. Touches the button (opens the door)
3. Heads straight to the goal (steepest gradient when door is open)

No planner told it to do this.

Scaling Up: Blocks World

AICON solves the classic Blocks World puzzle — rearranging block towers into a goal configuration — a benchmark that normally requires BFS or A*. The state is a matrix o[X,Y] (likelihood X is on Y) and vector c[X] (likelihood X is clear):

# Eq. 5: Is block X clear?
c[X] = 1 - (1/n) * sum(o[Y, X] for Y in blocks)

# Eq. 6: State update
o_new[X,Y] = o[X,Y] + c[X] * c[Y] * a_stack[X,Y] - c[X] * a_unstack[X,Y]

The gradient of the goal cost through these equations automatically decides whether to stack or unstack, and which blocks, and in what order (aicon_policy.py):

def _select_action(self, o, c, goal_cost):
    best_action, best_norm = ("stack", 0, 1), -1.0
    for X in range(n_blocks):
        for Y in range(n_blocks):
            if X == Y: continue
            if c[X] > 0.5 and c[Y] > 0.5:        # ∇_stack (Eq. 7)
                o_new = (o + c[X] * c[Y] * a_stack).clamp(0, 1)
                grad_norm = abs(goal_cost - goal_cost_fn(o_new))
                if grad_norm > best_norm:
                    best_action = ("stack", X, Y); best_norm = grad_norm
            if o[X,Y] > 0.5 and c[X] > 0.5:      # ∇_unstack (Eq. 8)
                o_new = (o - c[X] * a_unstack).clamp(0, 1)
                grad_norm = abs(goal_cost - goal_cost_fn(o_new))
                if grad_norm > best_norm:
                    best_action = ("unstack", X, Y); best_norm = grad_norm
    return best_action

Result: 100% solve rate on 130 randomly generated instances with 10–30 blocks, including tasks requiring 35+ steps. Matches optimal BFS in step count. No forward search.

Real-World: Drawer Manipulation

A Franka Panda with a wrist camera and force-torque sensor opens a physical drawer under uncertainty. Four components, two interconnections:

When Σ_drawer is large (high uncertainty), the steepest gradient path goes through reducing uncertainty — “move the camera to look around.” Once Σ_drawer shrinks, the gradient shifts to “reach and grasp.” After p_grasped flips to 1, it becomes “pull.” The robot sequences the entire task from one cost function: g(q) = (q − 0.20)².

Results vs. Planning Baselines

70 real-world trials: 7 conditions × 10 runs each. Baselines: state-space planner (fixed trajectory) and 67-dimensional belief-space planner (3 viewpoints + 2 kinematic explorations).

AICON handles both because it has no plan to become stale — the gradients recompute from live sensor feedback at every step. Under high uncertainty, it spontaneously invents triangulation — nobody programmed that behavior.

Ablations: Why the Gates Matter

Remove p_visible or p_grasped and performance craters. Sum all gradients instead of selecting the steepest and you get jerky, unstable motion. The steepest-gradient selector is crucial because competing subgoal gradients point in opposite directions and canceling them produces garbage.

My Take

What’s genuinely impressive:

• The math is minimal (sigmoid gates + chain rule + argmax) but the emergent behavior is sophisticated
• Zero-shot: no training, no reward shaping, no demonstrations
• Disturbance recovery is automatic — there’s no plan to invalidate, just gradients to recompute
• Matching BFS optimality in Blocks World with pure gradient descent is a striking result

Honest limitations:

• The graph structure is hand-designed per task. You need domain knowledge to pick the right components and gates
• Gradient path enumeration scales poorly. The drawer has ~12 paths; a kitchen scene would have thousands
• Everything must be differentiable. If your sensor model isn’t a PyTorch function, you’re stuck

Why it matters for this class:
Most current robotic AI either learns a policy end-to-end (which requires massive data) or uses a classical planner (which requires a precise world model). This paper suggests a third path: if you can express your world model differentiably, gradient descent handles task sequencing for free. That’s a structurally important insight, even if the current framework is limited to structured domains.

Code

Pure PyTorch, no simulator required for the demos:

git clone https://github.com/poad42/no_plan_everything_control
cd no_plan_everything_control

# Locked door (2D, instant, matplotlib)
python scripts/simple_demos/demo_2d_locked_door.py

# Blocks World solver
python scripts/run_blocks_world.py

Paper: arXiv:2503.01732
Supplementary videos: tu.berlin/robotics/papers/noplan

I disagree.

Call it: backporting FP8-style numerics experiments to the RTX 3090.

Not because Ampere magically does FP8 compute (it doesn’t), and not because this makes an RTX 3090 “faster” than Hopper (it won’t).

But because a lot of FP8 research and engineering is really about:

• how you store weights (bytes on the wire)
• when and where you expand them (decode)
• what scaling/quantization contract you enforce

You can explore a surprising amount of that on consumer Ampere, if you’re willing to treat FP8 as a storage format and map the math onto hardware that is available.

Quick note: if you see an acronym you don’t recognize, jump to the glossary.

Code: https://github.com/poad42/cuda-fp8-ampere

The plan

Ampere (sm_86) has extremely capable tensor cores, but it doesn’t have native FP8 tensor-core MMA. What it does have is a very fast path for INT8 tensor cores (IMMA / WMMA).

So the project becomes:

Keep weights stored as 1-byte FP8 bit patterns in VRAM, decode/scale/quantize on the fly, and use INT8 tensor cores for the matmul.

That’s the whole framing: democratize FP8 research by making the storage + numerics experimentable on hardware people actually have.

FP8-as-storage, in one paragraph

I am not trying to do “FP8 compute.” I’m trying to store weights in a compact FP8 format and only expand them when needed.

The VRAM part is simple: FP16/BF16 weights cost 2 bytes/weight, while FP8 weights cost 1 byte/weight. So for large weight matrices, storing FP8 can cut the resident weight footprint (and the bandwidth to stream it) by close to 2×.

In practice you also store scale factors (e.g. one FP16 scale per output channel), but that overhead is tiny compared to the full $N\times K$ weight matrix.

Conceptually:

1. Store weights as FP8 bytes (E4M3) — literally uint8 bit patterns.
2. Decode FP8 → FP16 on the fly using a 256-entry LUT.
3. Apply per-output-channel (per-column) scale.
4. Quantize to INT8 so the tensor cores can consume it.
5. Run IMMA (INT8×INT8→INT32 accumulate), then write FP16 output.

That’s the whole “FP8 without FP8 MMA” idea.

What’s actually new here (and what isn’t)

Three honesty bullets up front:

• This is not a claim that Ampere beats BF16/FP16 cuBLAS. In fact, for pure compute, cuBLAS is usually hard to beat.
• This is not full FP8 training. There’s no backward pass here.
• This project focuses on FP8(E4M3) storage. Extending to E5M2 is conceptually similar (another decode path), but I didn’t build it into this writeup.

So what is interesting?

Bit-level FP8 handling (LUT decode)

I store FP8 weights as raw uint8 bit patterns and decode them with a 256-entry LUT. Since there are only 256 possible FP8 bytes, decode is conceptually:

• u8 → fp16 via LUT[u8]

No __byte_perm tricks here — it’s mostly about making that decode cheap enough to hide behind the tensor-core pipe.

Scaling + quantization as a first-class contract

The weights aren’t “just FP8.” They’re FP8 bits + per-output-channel scale. The kernel makes that explicit: decode → apply scale → saturating quantize to int8 → IMMA.

Stochastic rounding (SR): important, but not implemented here

If you’re interested in FP8 training dynamics, stochastic rounding matters a lot. This project doesn’t implement SR (no backward pass), but if I were pushing this toward “training-like” experiments on older GPUs, SR would be near the top of the list.

Glossary (quick definitions)

• FP8(E4M3): an 8-bit float format. Great for storage, not great for high-accuracy math.
• MMA: matrix multiply-accumulate (the tensor core instruction family).
• IMMA / WMMA: NVIDIA’s tensor core path for int8 matrix multiply (instruction path / CUDA API).
• cuBLAS / cuBLASLt: NVIDIA’s GPU linear algebra libraries (GEMM).
• cp.async: an Ampere instruction to asynchronously copy from global memory to shared memory.
• l2pin: using “persisting L2” cache hints to keep hot tensors resident longer.
• Per-column scale: one scale factor per output channel; common in quantized inference.
• LUT decode: since there are only 256 FP8 bit patterns, decode can be a table lookup.

The pipeline, in one diagram

A (fp16/bf16)                B (uint8 fp8-e4m3 bits)         col_scales (u16 bits)
[M,K] row-major              [N,K] (represents KxN col-major)      [N]
      |                               |                               |
      |                               | (LUT in __constant__)        |
      |                               v                               |
      |                        fp8 -> fp16 decode                     |
      |                               |                               |
      |                               +-----------(per-column)--------+
      |                                           scale
      |                               |
      |                               v
      |                        fp16 -> int8 (sat)
      |                               |
      +--------------- int8 A --------+
                      (act quant)
                                      |
                                      v
                            WMMA/IMMA (int8) accumulate (int32)
                                      |
                                      v
                             D (fp16) written as [N,M]
                             (represents MxN col-major)

If you’ve never written CUDA kernels: that diagram is basically the whole story.

Baseline: PyTorch decode + matmul

Before writing any custom kernel, I wanted a baseline that matches the real workflow:

# weights stored as FP8 bytes
B_u8 = ...  # [N,K] uint8

# decode fp8 -> fp16 every iteration
B_fp16 = LUT[B_u8] * scales[:, None]

# compute in fp16 using standard matmul
out = A @ B_fp16.T

That’s the easiest FP8-as-storage implementation: store FP8 bytes, decode on demand, then use cuBLAS.

Two additional baselines are useful:

• Decode + matmul + downcast output to FP8: what the pipeline looks like if you want to store the output/activation in FP8.
• Matmul-only with fp16 weights cached: not apples-to-apples (you’re no longer storing FP8), but it’s a useful upper bound.

Baseline numbers (PyTorch)

Measured on RTX 3090 Ti (sm_86), CUDA-visible, shape $M=N=K=4096$.

Notes (important, and easy to misread):

• The “matmul only” baseline assumes fp16 weights are already resident. That defeats the FP8 VRAM savings.
• “Peak alloc” here is per-call peak allocated bytes; it does not include already-resident fp16 cached weights.

The naive decode+matmul being fast is not a paradox — cuBLAS is extremely optimized, and the decode step is embarrassingly parallel. My main motivation for the fused kernel is controlling memory traffic and keeping the pipeline “weight storage = FP8 bytes” end-to-end.

Fusing it into one kernel

Once you accept that IMMA wants int8 fragments, the kernel is a pipeline problem:

• Where does decode happen? (constant memory LUT vs texture vs global)
• Where does scaling happen? (apply scale in fp16, or bake it into an int8 conversion)
• Where does activation quant happen? (register path vs shared-memory staging)
• How do you feed tensor cores continuously? (avoid stalls from decode/scale/quant)

I ended up implementing variants as a way to test hypotheses.

Variants (experiments)

• v2: baseline fused path (FP8→INT8 JIT + IMMA). Keep it simple and measure.
• v2_i8lut: “what if I precompute a per-column FP8→INT8 table in shared memory?” (sounds clever; didn’t win).
• v3_act_f16: fused activation quantization, register path.
• v4_act_f16: cp.async staging for activations + shared-memory quantization, then IMMA.
• texscale: load per-column scales via TEX.
• l2pin: persisting-L2 hints for B/scales.

Kernel benchmark numbers

Measured via ./build/gpu_bench on RTX 3090 Ti (sm_86), driver 590.48.01, CUDA 13.1.

Shape: M=N=K=4096, --warmup 10 --iters 50.

Notes:

• *_l2pin can vary with driver/GPU state and other workloads.
• The int8gemm cuBLASLt number is not FP8-as-storage; it’s a ceiling for int8 TC GEMM on this machine.

Why do this at all?

After seeing the tables, the fair question is:

If naive Torch is already fast, why bother?

Because “fast” depends on what you’re measuring.

On pure matmul throughput, a highly tuned fp16/bf16 GEMM can absolutely win. This project is about a different constraint: weight storage and weight movement.

If your weights are truly stored in FP8 (1 byte/weight), then compared to fp16/bf16 (2 bytes/weight) you’re targeting up to 2× less weight traffic and 2× lower resident weight footprint. That’s a real lever for memory-bound inference workloads — even if you pay some extra compute to decode/scale/quantize.

Practically, the “democratizing FP8 research” win is:

• you can keep the storage format honest (FP8 bytes in VRAM)
• you can experiment with scaling/quantization contracts
• you can measure the cost of decode/quant instead of hiding it in a pre-processing step

So I view this as a tool for exploration: FP8-as-storage end-to-end, on hardware that doesn’t officially “support FP8.”

Try it yourself

Repo: https://github.com/poad42/cuda-fp8-ampere

Build:

git submodule update --init --recursive
cmake -S . -B build -DCMAKE_BUILD_TYPE=Release
cmake --build build -j

Run tests:

cd build
ctest --output-on-failure

Run the kernel benches:

./build/gpu_bench --bench imma_fp8_jit_v2 --M 4096 --N 4096 --K 4096 --warmup 10 --iters 50
./build/gpu_bench --bench imma_fp8_jit_v4_act_f16 --M 4096 --N 4096 --K 4096 --warmup 10 --iters 50
./build/gpu_bench --bench imma_fp8_jit_v4_act_f16_texscale --M 4096 --N 4096 --K 4096 --warmup 10 --iters 50

Run the Torch baselines (including end-to-end downcast):

. .venv_torch_cuda312/bin/activate
python scripts/bench_torch_vs_fp8imma.py --M 4096 --N 4096 --K 4096 --kChunk 32 --report_mem --downcast_out_fp8

Next steps

If I had another weekend:

• Add a tiny numerical correctness harness (reference decode + GEMM with tolerances).
• Report a more honest memory metric: resident weights + peak workspace, not just per-call peak alloc.
• Try more realistic shapes (transformer-ish M, larger N, varying K) instead of only 4096³.

If you want to dig into the code, the repo contains:

• a CUDA kernel library (C++ API + C ABI)
• a benchmark harness (gpu_bench)
• a minimal PyTorch extension
• smoke tests (CTest + torch compile/import test)

If you look at the spec sheet for the Rockchip RK3588 (the chip inside the Orange Pi 5), it looks like a beast. It promises 6 TOPS of NPU performance. For $100, that’s a steal.

But if you try to run modern AI on it—specifically the Vision Encoder from SmolVLM—that promise falls apart.

The standard Computer Vision SDK (rknn-toolkit2) is optimized for older, predictable CNNs (like ResNet). When I fed it the SigLIP Vision Transformer used by SmolVLM, the driver choked. Even though the model is “smol,” the massive Attention matrices it generates triggered cryptic hex errors and refused to compile.

This left me with one option: running the model on the CPU. The result? A single image inference took ~30 seconds. The 6 TOPS accelerator sat idle while the CPU struggled.

I didn’t accept that. I decided to reverse-engineer the NPU to find out exactly why it was failing, and how to force it to run at full speed.

Context: Why do it the hard way? (First Principles)

A quick note for those following the ecosystem: You might see projects like QEngineering running the newer SmolVLM-v2 on Rockchip’s rknn-llm SDK.

That approach uses a specialized “black box” toolchain designed specifically for Transformers. Rockchip engineers have likely already implemented complex memory management inside that SDK to handle these models.

My project targets the original SmolVLM-v1, but more importantly, I built it on the legacy rknn-toolkit2 stack. Why hack the legacy stack? I wanted to take a “First Principles” approach. I didn’t want to use a black-box solver. I wanted to understand why the hardware was crashing on Attention layers and if I could find universal architectural patterns—like manual tiling and graph sharding—that could force any Transformer to run on any constrained edge accelerator.

The Detective Work: What is Error 0xe010?

Rockchip doesn’t publish a public Instruction Set Architecture (ISA). When I tried to compile the Attention layers, the driver kept spitting out an undocumented error: REGTASK Overflow (0xe010).

I hypothesized this was a memory overflow. Even though the model parameters are small (~96M), the intermediate activation matrices for a 1024-token sequence are huge (~25MB).

I wrote a script to generate synthetic ONNX graphs to probe the hardware limits:

• 8KB Tensor: Pass.
• 16KB Tensor: Pass.
• 32KB Tensor: Pass.
• 32.1KB Tensor: CRASH.

Discovery: The NPU has a hardware-enforced 32KB L1 SRAM Scratchpad for vector operations.

The standard compiler was trying to shove a 25MB Attention matrix into a 32KB slot.

The Fix: Nano-Tiling & The “Poison Pill”

To solve the 32KB limit, I wrote a “Nano-Tiling” algorithm in PyTorch. I manually sliced the massive 1024-token sequence into tiny 32x32 tiles that fit perfectly into the 32KB scratchpad.

But here is where it got messy. The rknn compiler is “smart.” It looked at my tiled graph, decided it was inefficient, and fused the operators back together into a single giant block… which immediately crashed the hardware again.

I had to trick the compiler. I needed a way to tell it: “Do not merge these nodes.”

I introduced a topological barrier I call the “Poison Pill.” I injected a dummy operation that looks mathematically significant to the dependency graph (preventing fusion) but is mathematically irrelevant to the model output.

# The "Poison Pill"
# 1. Take a slice (forcing a strided access)
slice_x = x[..., :1]

# 2. Apply a non-linear op (breaks compiler fusion heuristics)
# 3. Scale it down to near-zero so it doesn't affect the math
poison = torch.sigmoid(slice_x) * 1e-6 

# 4. Inject dependency
# The compiler sees 'out' depends on 'poison' and creates a barrier.
out = out + poison

By injecting this into the graph, I successfully forced the compiler to respect my tiling logic.

The “SigLIP Cliff”: Solving Accuracy Collapse

Getting it to run was step one. Getting it to be right was step two. When I first got the NPU running, the output was garbage. The cosine similarity compared to the original model was 0.02 (pure noise).

The culprit was the architecture of SigLIP. Unlike standard models, SigLIP has massive activation “spikes” (values around 300.0) sitting next to tiny visual signals (values around 0.05).

NPU quantization (INT8) works by mapping the range to -128/+127.

• If you zoom out to capture the 300.0, the 0.05 rounds down to 0. Signal lost.
• If you zoom in to capture the 0.05, the 300.0 overflows to infinity. Math crash.

I implemented a “Sandwich” Domain Shift:

1. CPU Pre-Scale: Multiply the input by 0.1. Now the max value is 30.0 (Safe for FP16).
2. NPU Execution: Run the heavy compute in this scaled-down “safe zone.”
3. CPU Post-Scale: Multiply the output by 10.0.

This simple trick restored the signal fidelity from 0.02 to 0.999 (effectively bit-exact).

The Architecture: Custom Runtime Scheduler

Finally, to bypass driver timeouts caused by the sheer number of tiles (thousands of tiny operations), I physically cut the model graph into 26 separate binary files (shards).

I wrote a custom User-Space Runtime in Python that acts as an orchestrator. It manually loads these shards onto the RK3588’s 3 separate NPU cores and fires them in a synchronized round-robin schedule (Core 0 -> Core 1 -> Core 2).

The Results

By ignoring the vendor’s “Unsupported” warnings and re-architecting the software to match the silicon’s physical reality, the results were drastic.

Conclusion

This project challenged the binary notion of “Supported Hardware.” The RK3588 didn’t support the SigLIP encoder out of the box on the standard SDK, but the silicon was always capable of it. It just needed an engineer to dig into the register overflow codes and manage the memory manually.

If you want to see the full code, including the tiling logic and the runtime orchestrator, check out the repo below.

View Source on GitHub