Triple frame buffer FPGA implementation

This hardware can primarily serve as a frame buffer for a simple camera setup. As the camera transmits frames, the frame buffer reads the frames into a buffer, and later writes the frames out to another piece of hardware such as a microcontroller or CPU. The flexibility of FPGA fabric allows our buffer design to be tailored to and integrated with a user’s project while adding as little overhead as possible.

Frame buffers have been in use since Noll at Bell laboratories [1] used a frame buffer to implement a graphical display on a Honeywell DDP-224 mainframe computer. As the need for graphics processing and graphical user interfaces increased, double and triple frame buffers were implemented. Double buffers have the advantage of allowing both processes to access at least one buffer at a time, however one process (i.e a camera or a display) will need to wait until the other has finished with the buffer before a swap can occur. Triple buffers avoid this waiting problem by always having an extra buffer available so that one process can still use one buffer uninterrupted while the other swaps buffers. This effectively decouples the processes from one another and allows them to run independently. Double buffering can also be used in applications which are not strictly graphical such as DMA to meet the addressing requirements of a device. These buffers are referred to as “bounce buffers” in UNIX [2]. One example of a proprietary double buffer can be seen in Nvidia’s patent [3] where two logical buffers are used to buffer image data between a system bus and a display. Oracle also holds a patent [4] on a frame buffer system which utilizes a similar double buffer scheme. Nvidia has used triple buffering to implement a software-side rendering speedup known as “Fast Sync” [5]. All of these examples demonstrate special purpose buffers built to suit the needs of a specific system. Without using re-programmable logic, it is difficult to build an adaptable buffer using off-the-shelf components. By using an FPGA to implement the buffer, the design can be customized to fit within the confines of a given project. To the extent of the knowledge of the authors, a generalized open-source verilog-based triple frame buffer has yet to be presented.

Many examples of FPGA-based image processing techniques are already available [6], [7], [8], [9], [10]. Johnson et al. [9] detail several techniques for creating image processing designs, the majority of which used pipeline approaches. Hiraiwa et al. [8] demonstrate a real-time computer vision architecture which uses an FPGA-based pipeline to implement complex vision algorithms. Greisen et al. [10] demonstrate a high definition stereo video processing and correction pipeline whose front end is implemented using an FPGA. Battle et al. [6] present an image processing technique which uses a parallel pipeline architecture consisting of reprogrammable processors to implement image processing algorithms. Unzun et al. [7] present a fast Fourier transform algorithm implemented in an FPGA using a pipeline approach. The common theme between these image processing techniques and many others is that they used a pipeline approach in which data is processed on a continuous basis. This makes these algorithms excellent candidates for integration with our frame buffer. These algorithms could be used as a replacement for the VGA controller in our reference design, allowing them to take full advantage of the reduced latency and frame loss. The process of integrating algorithms similar to these into our design will be covered further in 6.3. Some examples of open source FPGA-based buffers exist [11], [12], however these are tailored for a specific purpose such as HDMI video buffering or power saving. Many FPGA projects such as [13], [14] include various buffer stages, but fail to provide any insights into their design.

Some of the source files for this project were originally obtained from [15]. Their implementation makes use of a dual-access memory buffer implemented using FPGA resources. While this method is faster and simpler than our triple buffer approach, it requires vast amounts of device resources to construct the memory, and thus is not feasible for smaller devices or very large buffers which would be better implemented as SRAM or SDRAM buffers for cost saving purposes. While many modern microcontrollers include features such as DMA which allow a frame buffer to be implemented [16], these implementations suffer from delay caused by the inherent overhead imposed by the microcontroller along with any overhead incurred from the DMA subsystem.

This implementation consists of a proprietary buffer controller module, three custom multiplexer (mux) modules, and several peripheral modules used for capturing and transmitting data. All modules from the authors are written in Verilog and are completely customizable to enable users to change bus width, frame eviction policies, and interface protocols to fit their project needs. Some modules are sourced from other projects [15]. Compared to previous methods such as ASIC or software implementations, this implementation has two advantages. While ASIC implementations are immutable and cannot be modified after they are put into silicon, our Verilog implementation allows for hardware specifics to be easily modified to suit the user’s needs. While software implementations are arguably more customizable than HDL implementations, their speed and performance will be far less than that of an HDL-based implementation due to the execution overhead. This implementation can also serve as a starting point for more complex implementations which may include error checking, flow control, or other capabilities.

When operating the provided reference implementation, the OV7670 must first be connected to the DE2-115 development board via the GPIO interface. The exact pin connection specifics can be found in the pin planning window of the reference Quartus project, and the pinout of the GPIO header can be found on the DE2-115 website [20].

The next step is to connect the two external SRAM chips. Our reference implementation uses the chips found here: [18]. These two chips should be connected to the FPGA via the HMSC expansion header. The third required chip is already on the board. A standard 40-pin breakout for this header can be found here: [21]. A complete list of the pin connections can be found in the Quartus pin planning window for the project along with the data sheet for the breakout board. It should be noted that the “lb_a_n”, “ub_a_n” and “ce_a_n” pins for both external SRAM chips should be connected to ground rather than the FPGA.

Button #0 is configured to be the reset button for the frame buffer. This will reset all internal states within the frame buffer along with reconfiguring the camera via the I2C interface. In the top level design simply called “TripleBuffer”, Button #1 will reconfigure the camera without resetting the system. The green LED #0 is the “config_finished” LED which denotes when the FPGA has finished configuring the camera.

To ensure that the buffer handover is working properly, the “transmission_trigger”, “capture_trigger”, and “sram_select” signals should be probed via SignalTap along with the internal “x_cnt”, “y_cnt”, and “z_cnt” registers as well. The SignalTap file is already set up within the reference project, however it is disabled by default. After enabling Signal Tap and enabling the aforementioned signals, the buffer handover procedure can be validated by ensuring that every time one of the trigger signals becomes active and the counter associated with the available buffer is less than or equal to 2, the “sram_select” signal changes, indicating that one of the buffers has been switched into a new SRAM chip. For the case in which one of the counters is 2 or greater, it should be ensured that when the “transmission_trigger” goes active, the buffer controller does not change its “sram_select” signal and prevents the handover from occurring.

These signals can also be used to measure frame rate. The frame rate for this device assuming that each frame is a new frame can be taken to be the rate at which the “sram_select” signal changes state immediately after the transmission trigger becomes active. Simply measuring the rate at which the “sram_select” signal changes will provide an inaccurate measurement as the signal also changes when the camera has finished writing a frame. The actual frame rate for this device, or the frequency at which a frame is sent to the VGA controller, is defined by the rate at which the VGA vsync signal becomes active.

As mentioned in Section 4.2, the address and data bus sized can be modified by explicitly providing parameters for these values in the module declaration. This allows for the MUXes to be scaled to fit any design. While increasing the bus widths will change the sizes of the MUXes and thus change the number of logic elements needed, the size of the data controller will not change. The largest buffer that could be created with this hardware is only a function of the memory modules, not the control hardware. For example, on the DE-115 board, one could use the HSMC expansion header to add 2 additional 19-bit SRAM chips, bringing the total buffer storage to 1.5 MB (500 KB per buffer).

While the authors will provide relevant timing information regarding the example design, it is important to note that timing requirements can vary greatly across various platforms. For example, high-end FPGAs will be able to run this design at much higher speeds while still meeting timing requirements while cheap and widely available FPGAs may struggle to achieve timing closure at faster clock speeds. When operating larger designs, it is likely that the critical paths will be found in modules added by the user. This is because the buffer logic does not contain data paths long enough to dominate the timing requirements in most situations. This is the case for the presented example design as the longest delay path is found in the VGA signal generation logic.

This design can be properly constrained with an input clock no faster than 167.5 MHz under the current operating conditions of the camera and VGA module (both running at 25 MHz) and the current hardware. It may be possible to still constrain the design at higher clock frequencies by increasing the clock frequencies of the camera and VGA signal generator.

An additional reference design has been provided which showcases a usage case in which the incoming data stream is significantly faster than the outgoing data stream. In the design, the capture buffer has been replaced by a UART receiver while the transmission buffer has been replaced by a UART transmitter which operates 8 times as fast as the receiver. The buffer controller allows the two to exchange data while allowing both UART controllers to swap buffers independently.

In the design, the UART transmitter is gated by a control signal to showcase the importance of the independent buffer exchange. If the UART transmitter did not have any restrictions on when and how fast it could transmit, then a triple buffer system would be unnecessary. However, because the transmitter can be delayed unexpectedly via the control signals, it is possible that the transmitter would not be finished with a buffer by the time the receiver had finished with its buffer, which would force the receiver to wait for the transmitter. With three buffers, the receiver would be free to switch to the third buffer while the transmitter finishes up its buffer and switches on its own time.

The authors of this paper have no conflicts of interest involving any of the components or technologies used in this project. We have not been paid by anyone to publish this work. This research did not receive any specific grant from funding agencies in the public, commercial, or not-for-profit sectors.