Building an FPGA accelerator and Its Compiler

In my previous MLIR compiler project, I built a small tensor compiler using the MLIR framework. While working on it, I realized how much complexity the framework handles for me, from lowering operations through existing compiler infrastructure to generating executable code.

This made me curious about what happens underneath these abstractions, particularly when compiling for custom hardware rather than a conventional CPU.

For this project, I want to explore two things: how AI accelerator architectures work and how a compiler backend generates instructions for custom hardware.

I'm building a small FPGA-based accelerator alongside a compiler backend written in C++17 using only the standard library. There won't be a frontend for now, since my main focus is understanding compiler backend algorithms, instruction generation, and hardware-aware optimizations.

The project starts with a simple scalar execution unit, which is already operational, and will gradually evolve to incorporate a tensor unit and on-chip memory buffers. I've also developed a C++ compiler backend that generates instructions for the custom hardware, with successful end-to-end execution on the FPGA. Along the way, I want to measure how specialized hardware and compiler-controlled data movement affect matmul performance.

This is primarily a learning project and the goal is to develop a better understanding of the relationship between compiler design and computer architecture through hands-on implementation.

Source code available on my Github

High-Level Architecture

The system is built around a Terasic DE10-Lite FPGA board, using Intel's Nios V/m soft-core processor.

The design includes a 50 MHz clock, a PLL for generating the clocks used by the SDRAM interface, external SDRAM for the processor's main memory, and a JTAG UART for communication with the host computer.

The Nios V/m processor loads compiler-generated instructions into the custom accelerator and controls its execution. The accelerator currently consists of a controller, instruction memory, a small register file, and a scalar arithmetic unit supporting addition, subtraction, and multiplication.

On the software side, the compiler transforms a small intermediate representation (IR) into instructions for the accelerator's custom instruction set.

Together, these components provide an initial end-to-end path:

Compiler IR -> Register allocation -> Instruction encoding -> Nios V/m loader -> FPGA Custom Unit -> Result

Following the project

I'll document the project incrementally, including the hardware architecture, compiler implementation, design decisions, experiments, and lessons learned.

To keep things manageable, I'll separate the detailed FPGA and compiler implementations into their own articles.

• Part 1: Initial FPGA architecture with custom scalar unit

• Part 2: Initial Compiler backend for custom scalar unit (write-up in progress)

What's Completed So Far?

• Built the initial FPGA system with Nios V/m, external SDRAM, JTAG UART, and custom accelerator controller

• Implemented a scalar execution unit supporting integer addition, subtraction, and multiplication

• Added hardware testbenches, including directed and randomized testing

• Built a C++ compiler backend skeleton with IR validation, register allocation, instruction encoding, and a reference executor

• Successfully executed compiler-generated instructions on the FPGA and verified the results against software execution

What's next? (currently planned, I may add, remove, or modify based on situation)

• Improve the compiler and FPGA testing workflow with automated and randomized test cases

• Implement liveness-based register reuse in the compiler

• Explore compiler correctness through symbolic execution and SMT-based translation validation using Z3

• Study existing AI accelerator architectures to improve future design decisions

• Implement scalar matmul as the performance baseline

• Add a tensor unit and extend the compiler to generate instructions for it

• Explore instruction selection and dependency-aware scheduling for the expanded ISA and hardware architecture

• Compare naive tensor execution against the scalar baseline

• Explore compiler-controlled tiling and on-chip buffer reuse to improve data movement and performance

• Potentially introduce a cost model for selecting tiling configurations

Curent Scope and Limitations

The project currently focuses on integer arithmetic and will initially use matmul as the primary workload for evaluating hardware and compiler optimizations.

While matmul is an important operation in AI workloads, it doesn't represent every computational pattern. Other operations, such as elementwise computations, reductions, and normalization, introduce different challenges in scheduling, memory access, and execution.

For now, I'm keeping the scope focused on matmul and compiler-controlled data movement, with the possibility of exploring other operations and optimizations, such as operator fusion, in future iterations.