Libcamera CCM tuning

This is a note about what I did for the image quality tuning on a Raspberry Pi.

Photo by Nejc Soklič on Unsplash

First of all

The following components have been used for verification.

• SoC: Raspberry Pi CM4
• Sensor: IMX219
• Lens: JSD2629
• Library: Libcamera

Tuning parameters are limited to CCM, and image quality (DR, sharpness) is outside the scope of verification. The Libcamera config.json is used to tune CCM, but the config.json should already be calibrated for the sensor and lens.

This time, we did not fully understand the true meanings of parameters but we arrived at our findings by induction. The information is not always documented somewhere and is not guaranteed to be correct, as the parameters were changed and the results checked repeatedly.

Background

I have been involved in the development of a camera-based IoT project for the last two years, and as it is a consumer product, I want to be particular about the image quality and have tried several times in the past, including with my team members, to create my images, which have failed. There are two main reasons for this failure: firstly, image creation tends to be subjective. The second reason is that the variables involved in the image process are very complex and unsystematic. However, it was impossible to find anyone who understood which parameters had to be tweaked to achieve the desired image.

In the meantime, we were able to find a member of the outsourcing team who found this situation interesting. So, together with him, I tried again to create a picture and could systematize it to some extent, so I wanted to record this knowledge.

What I did

Quantify the difference

First, it was decided to quantify the CCM changes in order to objectively determine whether they have been implemented correctly. In addition, it was decided to find a benchmark to make the differences easier to understand when quantifying them. In this case, we decided to use GoPro as a benchmark and aim to create the same pictures as GoPro, not because we have a strong preference for GoPro, but because we have one in our house and I thought it was a beautiful picture. In the future, when I found a better benchmark from GoPro, I was aware of the versatility of being able to tune to that benchmark picture.

For quantification, we decided to use the CIEDE2000 definition.

Color/Tone and Colorcheck Appendix

When I spoke to the engineer who had previously programmed the camera’s white balance, he said that a color difference of 1 or less was the same picture, so I set an objective target of a CIEDE2000 color difference of 1 or less for each color on the 24-color checker compared to the GoPro.

Prepare code snippets

We used the code in the following repositories to perform the measurements.

GitHub - atsss/RaspberryPi_Camera_IQ_Tuning

A description of each file is given below.

• raspberrypi/server_1gain.py : To start the server to take the original image without correction.
• raspberrypi/server.py : To start a server to take images with its own CCM applied.
• laptop/color_accuracy_V2.py : To display the difference between the target and the captured image.
• laptop/color_checker_V2.py: To generate CCM reflecting the result of modifying parameters.

Code lines for parameters are given below.

https://github.com/atsss/RaspberryPi_Camera_IQ_Tuning/blob/fb09cc427b4fffa6b491b0da29394cfb682aeb50/laptop/color_checker_V2.py#L91-L106

Procedures

1. Connect to the Raspberry Pi via SSH, run server_1gain.py and take a photo with no correction applied by CCM.
   a. Check the local IP using ifconfig
   b. Open http://{local_ip}:8080/ in a browser and save the image
2. Run color_checker_V2.py on the laptop and copy the CCM in the log to imx_219.json
   a. Set the CCM to a value around the color temperature (CT) that the Raspberry Pi recognizes in the log when the server is run.
3. Connect to the Raspberry Pi via SSH, run server.py and take a picture using the CCM calculated in step 2.
   a. Same as 1.
4. Execute color_accuracy_V2.py on the laptop and modify the parameters in the following order while viewing the image comparison.
   a. Gain → Brightness
   b. Contrast → Color intensity
   c. Color temperature → Color of the light source
   d. RGB offset → Fine adjustment of R, G, and B respectively.
5. After manually modifying the parameters in color_checker_V2.py, run color_checker_V2.py on a PC and copy the CCMs from the log to imx_219.json
   a. Repeat 2 → 5 thereafter.

How to adjust gain

Brightness cannot be adjusted using the RGB difference values, so the brightness is adjusted. Increase the value to brighten and decrease the value to darken, applying the brightness component to the A, E and I of the CCM.

The point is to focus on the difference between the white and black colors. color_accuracy_V2.py is executed to create a PNG file comparing the color chart in a horizontal line. Visually check this PNG file to ensure that the upper and lower borders of these two colors are eliminated.

At this stage, the total color difference is 250 ~ 300.

How to adjust contrast

Contrast is manipulated to make color differences clearer. As the colors were generally the same, they are manipulated to be darker (or lighter): raising the A in the CCM and lowering the BC makes the contrast on the R side clearer. Similarly, the same values are put in on the G side and B side.

The point is to focus on the borders of colors other than black and white. color_accuracy_V2.py is executed to create a PNG file comparing a color chart in a horizontal line. Visually check this PNG file to ensure that the upper and lower borders of colors other than black and white are eliminated.

At this stage, the total color difference is 200 ~ 250.

How to adjust color temperature

Color temperature is manipulated to compensate for color variations caused by the light source. It is manipulated to add an overall red or blue color overlay: the color of the light source is multiplied by the A ~ I element of the CCM.

The point is to focus on the color of white. Running color_accuracy_V2.py creates a PNG file comparing a color chart in a horizontal line. Visually check this PNG file to see what color is on the white. In the future, it may be possible to calculate what color the top (target) white is and correct it. Currently, the color tones that are visually confirmed from the following website are selected and corrected.

What color is a blackbody? - some pixel rgb values

Note that the CT that the Raspberry Pi logs is not accurate, so do not correct the color temperature based on that value.

At this stage, the total color difference is 120 ~ 200.

How to adjust RGB offset

RGB offsets are manipulated to tweak differences of red, green, and blue colors separately. Do not change A, E, or I in CCM, but use other values to correct them. For the red color correction, correct the vertical readings such as D and G.

From here, you can directly correct the R, G, and B with the biggest color difference based on the color difference values that appear in the log. An example is shown below.

dark skin: 4.992451527064038
light skin: 7.995300240478907
blue sky: 1.5546189400406092
foliage: 1.9560340955557227
blue flower: 2.280947362960797
bluish green: 7.874029059612351
orange: 3.683277873136054
purplish blue: 2.7748941254007904
moderate red: 10.229388081200662
purple: 4.724487794331211
yellow green: 7.133206248393155
orange yellow: 3.529940257609023
blue: 2.442851128920905
green: 6.278710803289094
red: 8.892257851551868
yellow: 3.519098785052263
magenta: 5.40270346741261
cyan: 3.445895908735677
white: 3.805676027802727
neutral 8: 3.149572968155532
neutral 6.5: 4.963964866161974
neutral 5: 4.857752440727119
neutral 3.5: 4.092421908645695
black: 3.1965219728189473
Total: 112.77600373505774
Total: 4.699000155627406

In this case, it can be seen that the difference between the red colors is large. Therefore, modify r_offset. For each value, reduce the value if you want to strengthen the color, and increase the value if you want to weaken the color.

At this stage, the total color difference is between 100 and 120.

After all

this time the final total color difference is 109. Our best score is 102 following the steps. We can get less than 120 score constantly with these steps.

Based on the website, which is the description of color differences by a Japanese printing company, the 120 is in the Rank B and explained below.

The range of colors can be treated as the same color at the impression level. In the paint and plastics industries, color differences can lead to complaints.

「色を測る」ということ

We’re still aiming for less than 1 on each 24 colors. So I’m thinking it would be the next step to automate the steps and repeat them enough.

That’s it!