How to use OpenCV with Raspberry Pi CSI camera

by viilaamo.com

Update Jul 22, 2013, Maze fix is coming!

After a while with a long silence but now it seems that this maze issue will be fixed in the VideoCore4 ISP SW. It was indeed a green imbalance problem happening before the bayer processing as we suspected before. Fix is now in the releasing queue. We will verify the fix after we get our hands on the new software, which will be included in the RaspberryPi camera sw. Thank you James for fixing this!

Maze Pattern Issue

We found a possible CSI onboard camera/processing issue

When shooting towards the sun there is a maze pattern visible in flare areas. See the maze pattern sample (crop) below, and also click to open the original full size (JPG+RAW) image.

This maze pattern was clearly visible with both CSI cameras we have when shooting towards the sun. We have a lot of images showing this same issue. The horizontal blocks across the image are the unformatted bayer data (just for a reference), see the image below.

When zooming in more (image below), you can see that in the problem area every second row of green pixels are brighter than in the adjacent rows. This causes the maze pattern when processed with VC4, if demosaicing is done with OpenCV - there is no maze pattern.

Reference RAW and jpg images about the maze issue

This section shows few reference images about the problem.

Reference figure 1. The RAW image buffer stored from the Rasperry Pi CSI camera and stored in the png format. The image above looks green as there is no demosaicing done for this image. Click to open the full RAW image (15MB). You may download that image and zoom it to examine the Gr and Gb pixels differencies.

Reference figure 2. The RAW image buffer stored from the Rasperry Pi CSI camera and demosaicing done in the OpenCV side with the cvtColor() and then stored as a jpg image. The OpenCV debayering does NOT produce the maze pattern. However you can see some square effect because there is not done any noise filtering to this image.

Based on these findings we conclude that the root cause of the sun flare maze pattern is the RAW data Green pixel difference in every other row in the flare area. This issue should be discussed with the Raspberry Pi CSI camera manufacturer (OmniVision). There are few ways to hide this kind of problem with ISP processing, but it should be checked if the green imbalance could be fixed just by tweaking some manufacturer specific camera sensor settings.

Simple work around fix for the maze issue

Here is one simple way of fixing this green inbalance problem.


void maze_filter(vector<Mat> bgr_planes)
{
  // Green inbalance kernel, run to bayer data
  Mat kernGI=(Mat_<float>(3,3) << 0.125, 0.000, 0.125,
                                  0.000, 0.500, 0.000,
                                  0.125, 0.000, 0.125);
  // Filter the green level
  filter2D(bgr_planes[1], bgr_planes[1], bgr_planes[1].depth(), kernGI);
}

See results below; left is RAW image with just cvtColor debayered, middle is 2D filtered with the code above before debayering, right hand side shows the original JPG. Please note that there is no other image processing stages applied - hence the color difference compared to VC4.jpg image.

viilaamo.com

Raspberry Pi hackings...