FPGA real time Focus Peaking
Closed, ResolvedPublic

Description

To apply for this task please complete this mandatory coding challenge together with your application: T871
Applications without matching coding challenge completed will not be considered.


VHDL image pipeline logic to calculate gradients and generate focus peaking information for real-time visualization.

Goals:

  • Implement line buffer
  • Implement pipelined (sobel) kernel
  • Evaluate gradients for focus peaking

Prerequisites:

  • FPGA Know-How
  • Image Processing Know-How

Language Skills:

  • HDL (VHDL or Verilog)

References:

Mentors: Alex, Bertl

To get in touch with any mentor check the Mentor Contact List.

Related Objects

sebastian created this task.Feb 9 2017, 4:42 PM
sebastian updated the task description. (Show Details)
sebastian raised the priority of this task from to Normal.
sebastian moved this task to FPGA / HDL on the Google Summer of Code 2019 board.
sebastian added a subscriber: sebastian.
Bertl updated the task description. (Show Details)Feb 9 2017, 4:54 PM
Bertl updated the task description. (Show Details)Feb 9 2017, 5:22 PM
alex updated the task description. (Show Details)Feb 9 2017, 5:35 PM
sebastian updated the task description. (Show Details)Feb 9 2017, 6:16 PM

Hi,

I am a final year undergraduate student at National University of Sciences and Technology, with a major in Electrical Engineering. I participated in GSOC last year with TimeLab organisation and developed a time series simulator in Python.
I have a particular interest in parallel hardware programming. Recently, as a part of my semester project, I implemented a pipelined version of Convolutional Neural Network on FPGA using Verilog.

I am really interested in this project but I have a couple of questions that I would like clarified:

  • First, I am wondering if the mentors would be open to the implementation of other techniques (in addition to Sobel Filters) that are useful in film making?
  • Are we expected to port the algorithms on FPGAs to analyse performance?

First, I am wondering if the mentors would be open to the implementation of other techniques (in addition to Sobel Filters) that are useful in film making?

Definitely, what did you have in mind?

Are we expected to port the algorithms on FPGAs to analyse performance?

You mean try performance on different FPGAs?

  • I had in mind techniques that could be used for blurring, sharpening, un-sharpening, embossing, etc. I am not sure if any of these are already implemented, but my idea is to develop a common module in Verilog that can accept kernel of any size (e.g, 3*3 or 5*5) and produce the desired effect on image. For example, the user could input a kernel of size 5*5 to produce Gaussian blur.
  • Yes, I meant to quantify performance on different FPGAs.

I had in mind techniques that could be used for blurring, sharpening, un-sharpening, embossing, etc. I am not sure if any of these are already implemented, but my idea is to develop a common module in Verilog that can accept kernel of any size (e.g, 3*3 or 5*5) and produce the desired effect on image. For example, the user could input a kernel of size 5*5 to produce Gaussian blur.

A general kernel filter would make sense I guess, though I see no real use to blurring, sharpening or embossing effects in a cinema camera yet. But I am open to be convinced otherwise.

Yes, I meant to quantify performance on different FPGAs.

Our hardware designs are finished and the FPGA we use is decided so while its of course interesting to see differences the camera will use only the http://zedboard.org/product/microzed with Xilinx Zynq 7020.

Hi,
Here some of my queries regarding this project.
->What kind of edges does the camera require? What if the algorithm is developed if it only shows the edges when the camera is in motion? When the user is stationary it will store the edge information in memory, so that it will not have to do any processing.I think this may save computation time.

->Is the micro-zed connected to the CCD sensors() through its PMOD header?

->I was studying the " image processing pipeline paragraph" given in https://wiki.apertus.org/index.php/AXIOM_Beta/AXIOM_Beta_Software. I assume that the Sobel Filter to be designed will be in Image Overlay block of the pipeline?

Also, in addition to the above post, I would like to know about the various signals which the camera sensor (Truesense KAC12040 or Cmosis CMV!2000) data bus will provide?

Here is a link to a simple flowchart: https://github.com/sagniknitr/Real-time-sobel-filter-in-FPGA/blob/master/gsoc2.jpg

I would like to know the mentor's comment on this.

A useful resource for pipelined convolution implementation: https://daim.idi.ntnu.no/masteroppgaver/013/13656/masteroppgave.pdf

See section 4.4.4 (page 37) for the specific implementation.

Bertl added a subscriber: Bertl.Mar 23 2017, 1:29 AM

@sagnikbasu: Sorry for the delay, I obviously missed your questions.

Regardless of the source of the sensel data (Sensor, Memory, etc) it can be assumed that a number of consecutive sensel will be available for processing and the blocks will be fetched in a left-to-right and top-to-bottom order. The sensel need to be buffered for processing (if the algorithm requires to access previous data) and the output needs to happen in a similar way (i.e. pipelined and in some left-to-right, top-to-bottom order).

The amount of data which needs to be processed is quite large (4K @ 60+ FPS) so optimiziation and pipelining are key, decimation might also be an option.

The MicroZed is connected via up to 36 LVDS pairs with the CMOS Sensor. The Data is transferred in several serial streams.

Hope that clarifies,
Herbert

danuka added a subscriber: danuka.Mar 23 2017, 5:02 PM
Bertl renamed this task from FPGA real time Sobel Filter to FPGA real time Focus Peaking.Jan 23 2018, 12:18 PM
Bertl updated the task description. (Show Details)
Bertl moved this task from Qualification Tasks to FPGA / HDL on the Google Summer of Code 2019 board.
Bertl changed the edit policy from "All Users" to "Administrators".Jan 23 2018, 1:07 PM
sebastian updated the task description. (Show Details)Mar 8 2018, 8:24 PM

This is the paper that demosaics/ interpolates the pixel as well as preserve the edge information.:
http://ieeexplore.ieee.org/abstract/document/1703585

Cannot open it without logging in or subscription.

This link is open for all

sebastian closed this task as Resolved.Dec 22 2018, 1:09 PM
sebastian claimed this task.