Photographic Color Reproduction Based on Color Variation Characteristics of Digital Camera

(1)

This work was supported by a National Research Foundation of Korea Grant funded by the Korean Government (No. 2009-0077434).

DOI: 10.3837/tiis.2011.11.016

Photographic Color Reproduction Based on Color Variation Characteristics of Digital

Camera

Ui Seong Kim¹, Jong Min Lee¹, Yong Min Kim¹, Ki Tae Park² and Young Shik Moon¹

1 Department of Computer Science and Engineering, Hanyang University Ansan, Gyeonggi-do, 426-791, Republic of Korea

[e-mail: {uskim, jmlee, ymkim}@visionlab.or.kr, ysmoon@cse.hanyang.ac.kr]

2 General Education Curriculum Center, Hanyang University Ansan, Gyeonggi-do, 426-791, Republic of Korea

[e-mail: parkkt@cse.hanyang.ac.kr]

*Corresponding author: Young Shik Moon

Received July 13, 2011; revised October 12, 2011; accepted November 8, 2011;

published November 29, 2011

Abstract

In this paper, a new technique for color reproduction based on color variation characteristics of digital camera under a dim light condition is proposed. Generally, photographers should adjust a camera exposure properly for obtaining an image with real color tone of subjects. Thus, in case of taking a picture under a dim light condition, the exposure time of a camera has to be relatively longer than the one under a bright light condition. Although images with real color tone of the subject are obtained, the images may get blurred due to the shaky hands of photographer holding the camera. In order to avoid the blur effect, an input image is captured from a camera set as a short exposure time under a dim light condition. Then we propose a method to reproduce color tone of the dim input image. To this end, color variation characteristics which represent color variations of a digital camera are first extracted by analyzing the Macbeth color checker images taken under various exposure values. Then, a color reproduction is performed by an estimation based on the color variation characteristics.

Experimental results have shown that the proposed method has achieved better performance of color reproduction, compared with existing methods.

Keywords: Photographic color reproduction, color image enhancement, computational photography

(2)

1. Introduction

A

s the penetration rate of digital camera is increased by technical advances of hardware and software fields, it has been a part of our daily life to take a picture with the digital camera.

Furthermore, social networks such as personal blog, Facebook, and Twitter in the Internet make the digital camera the most popular one out of consumer electronic devices. However, althought many techniques related to a digital camera such as an auto-exposure (AE), an auto-focusing (AF), and a face detection have been developed so that non-experts can handle it effectively, it is not easy to satisfy demands of uers under various environments. In case of taking a picture under a dim light condition, the quality of images captured from a hand-held camera is limited because the amount of light reaching the image sensor is not enough. The intensity of a captured image is dependent on the exposure time. If the exposure time of a camera for obtaining the enough amount of light is long, an image blurred by shaky hands of photographer may be obtained. In contrast, if the exposure time is short, an image degraded in the color and the tone may be captured.

One method to prevent these problems mentioned above is that users should manually adjust various settings of a camera such as an aperture, an ISO sensitivity, and a flash light.

However, if the users are not experts, it is not easy to adjust the various settings properly. In case of applying a flash for taking images under dim light condition, harsh shadows can be appeared and color mood of original light can be disappeared due to the flash light [1].

The other method is to utilize a technique of computational photography. The computational photography is to converge fields of photography, computer vision, image processing, and applied optics, which enhances or extends the capabilities of digital photography [2]. There have been researched many methods of the computational photography such as a high dynamic range (HDR) imaging [3][4][5], a deblurring [6][7], a color enhancement [8][9][10], a flash/no-flash imaging [1], and a digital refocus imaging [11][12]. Recently, both a deblurring and a color enhancement out of image processing techniques have been studied actively.

In order to generally obtain an image with real color tone of subjects under dim light condition, the exposure time of a camera should be sufficiently long. However, since a blurred image caused by shaky hands of photographer or a motion of subjects can be obtained, it is needed to reconstruct a non-blurred image. A deblurring is to reconstruct an original image from a blurred image, which deconvolves the blurred image using a point spread function (PSF) describing the motion of a camera [13]. Although the deblurring method can reconstruct an original image accurately, ringing artifacts result from estimation errors of the PSF and additional devices are needed for estimating the PSF effectively. The computational complexity is also high.

A color enhancement is to compensate the light attenuation, and to restore a good color balance from a degraded image with low color contrast. This technique can be utilized to enhance a dim image captured by a short exposure time of a camera under a dim light condition. There are two methods such as a histogram equalization [8] and a retinex [9][10]

utilized for a color enhancement. A histogram equalization is a simple and effective method for enhancing a low color contrast because of extending the range between maximum value and minimum value by rearranging the distribution of color values. But, the histogram equalization results in false contours due to an excessive color change. The retinex based on the theory proposed by Land is to model the human visual perception, which observes the

(3)

sensations of color correlated with the degree of reflectance. This method first estimates the illumination by applying a set of Gaussian filters to an image with low color contrast. Then, an enhanced image is obtained by taking logarithm of the reflectance. However, results of the retinex may include halo effects or color fading depending on the size of Gaussian filters.

By using two defective input images, intensity correction method using Bayesian framework is proposed by Jia [14]. By combining two defective images which are an under-exposed image and an extended exposed image, a non-blurred and bright image is generated. The under-exposed image is dark and non-blurred. The extended exposed image is bright and blurred. However, this method not only utilizes two sequential images but also has heavy computational cost.

In this paper, in order to enhance a single under-exposed image, a new technique for color reproduction based on the color variation characteristics is proposed. The rest of this paper is organized as follows. Section 2 describes the definition of exposure value. The proposed method of color reproduction based on color variation characteristics of digital camera is presented in Section 3. Section 4 shows the experimental results, and conclusion is given in Section 5.

2. Exposure Value

In photography, an exposure refers to the amount light to reach an image sensor (or film). An exposure value (EV) is to combine a shutter speed and an f-number to be the same exposure [15]. The EV is defined as (1).

t EV N

2

log

2



(1)

where N is an f-number and t is a shutter speed [16]. The f-number is a ratio of focal length to current diameter of the aperture [17]. The f-number can be obtained by dividing focal length by current diameter of the aperture. f/1, f/1.4, f/2, f/2.8, f/4, f/5.6, f/8, f/11, f/16, f/22 out of f-numbers are available on common lenses. The more the aperture is opened, the more the f-number is decreased. The shutter speed denotes the time duration that the camera shutter is opened. The safe shutter speed is to be faster than the reciprocal of the focal length of the lens.

If the focal length of a camera lens is 50 mm, the shutter speed should be faster than 1/50 second in order to avoid the blur effect of an image [18].

The exposure is controlled by an aperture, a shutter speed, and an ISO sensitivity. The ISO sensitivity is used to measure the sensitivity of the image sensor. The image taken with the higher ISO sensitivity has more noises. The ISO 100 is regarded as the ISO sensitivity of the image sensor in this paper. Fig. 1 shows an EV table [17]. As EV is added by one, the amount of light reaching the image sensor is decreased by a factor of two times.

(4)

Fig. 1. Exposure value table: shutter speed in seconds on the horizontal axis and lens aperture in f-numbers on the vertical axis.

Fig. 2. Overall diagram of the proposed color reproduction method.

(5)

3. The Proposed Color Reproduction based on Color variation Characteristics

In this paper, by using a single under-exposed image, we propose a new color reproduction method based on color variation characteristics of a digital camera under dim light condition.

The proposed method is composed of two steps. In the first step, color variation characteristics are analyzed by utilizing images of the Macbeth color checker taken by a digital camera as an off-line processing. In the second step, as an on-line processing, by referencing the color variation characteristics and a difference (EV) between an EV of an under-exposed input image and a proper EV indicated by an exposure meter in a digital camera, the enhanced pixel values of the properly exposed image are estimated. Fig. 2 shows the overall diagram of the proposed method.

3.1 Analysis of Color Variation Characteristics of Various Camera Exposure Value

Color variation characteristics of a digital camera imply the degree of changing color pixel values on EV steps. In this paper, in order to estimate color pixel values of a properly exposed image, the color variation characteristics of the digital camera are analyzed from a non-blurred and under-exposed images. To this end, the Macbeth color checker including 24 colors which are scientifically determined is taken as source images for analyzing the color variations. Fig.

3 shows the studio environment for taking the source images. As shown in Fig. 3, we take a picture using a digital camera on a tripod in a studio with continuous lights fixed on the ceiling.

Fig. 3. The studio environment with continuous lights fixed on the ceiling and the Macbeth color checker.

The source images for extracting the color variation characteristics are taken by adding EV

(6)

by +0.5 on each image from a proper EV to +5 EV. The source images are taken 10 times at each EV and their mean is used for reducing the fluctuation of color pixel values. Therefore, the 24 dominant colors within the Macbeth color checker are obtained at each EV step. The samples of the source images are shown in Fig. 4. In Fig. 4, the top left image is taken by 7.5 EV, the next right image is taken by 8.0 EV, and the last image is taken by 12.5 EV. From each dominant color within the source images, we analyze the color variation of each R, G, and B channel. Accordingly, we generate 24 curves of color variation at each color channel. Fig. 5 shows the color variation characteristics of each R, G, and B channel at each EV step. As shown in Fig. 5, y axis implies the pixel value and x axis with 11 steps represents relative value compared to the proper EV at EV=0.

In this paper, we use the Canon EOS 550D digital camera with Canon EF-S 18-55 mm F3.5-5.6 IS lens in order to get source images for our study and a gray card is used to take an accurate exposure and remove the influences of illumination color.

Fig. 4. The Macbeth color checker images with various camera exposure values (f/5.6, 1/6 s to f/5.6, 1/180 s).

Fig. 5. Color variation characteristics of the Canon EOS 550D.

(7)

3.2 Estimation of Enhanced Color Pixel Values by Referencing Color Variation Characteristics

In this step, an enhanced image with proper exposure and color tone is obtained by referencing the color variation characteristics of each R, G, and B channel. To this end, we estimate colors to be enhanced from pixels in under-exposed input image by referencing the color variations of the Macbeth color checker images. Fig. 6 shows a method of estimating a pixel value for generating an enhanced image from an under-exposed image. As shown in Fig. 6, in order to estimate enhanced pixel values, we first find the intersection points on the curves at an input pixel value v_i, based on the indicated EV value. Then the enhanced pixel value v_o is considered as the weighted average of pixel values at EV=0 of the corresponding intersectional curves. If there are no intersectional curves, an upper curve and a lower curve close to the input pixel value are selected. The enhanced pixel value is estimated by (2).

1 1 2 2

2 1

1 2

1 2 1 2

, , v

o

w v w v

d d

w w

d d d d

   

 

 

(2) Where v₁ and v₂ are the pixel values of upper and lower curves at a proper EV, respectively and w₁ and w₂ are corresponding weights based on the distance d₁ and d₂, respectively.

Fig. 6. Mimetic diagram estimating an enhanced pixel value.

(8)

4. Experimental Results

In this paper, under a dim light condition, test images were taken with above the safe shutter speed in order to avoid an image blur. Consequently the test images have a higher EV than a proper EV indicated by an exposure meter in a digital camera. A difference between each EV of test image and the corresponding proper EV is also utilized in order to estimate enhanced pixel values.

In order to evaluate the performance of the proposed color reproduction method, we have used test images taken from the Canon EOS 550D digital camera with Canon EF-S 18-55 mm F3.5-5.6 IS lens under dim light conditions. In order to change an EV, after an f-number is fixed to f/5.6, a shutter speed is adjusted for taking test images. In our experimental environment, the shutter speed between 1/4 to 1/45 second for obtaining the test images with proper EV is used. In addition, the digital camera is set as auto white balance, evaluative photometry mode, high-quality large JPG with standard picture style and sRGB color space.

In each place for taking the test images, five images are taken with increasing EV by 1 from the proper EV indicated by an exposure meter in the digital camera.

Fig. 7 and Fig. 8 show test images and the corresponding results enhanced by the proposed method according to the increase of EV. For various EV, the proposed method shows that the results are stable and close to the image of the proper exposure. The results between Fig. 9 and Fig. 14 are the experimental comparisons of the proposed method and the existing methods such as color histogram equalization and multi-scale retinex with color restoration. For the comparisons, the test images taken with +4 EV from the proper EV are used. As shown in Fig.

9, the result of the proposed method shows that the colors of estimated pixels are very similar to the ones of an image with a proper exposure, whereas the existing methods could not estimate the colors of pixels properly. Fig. 10 shows that the result of the color histogram equalization is too bright compared with an image with a proper exposure and the result of the multi-scale retinex with color restoration is flat and is perceived strong halo effects. Our results in Fig. 11 and Fig. 12 are most similar to the proper exposed images whereas the colors enhanced by existing methods are different from the ones of the proper exposed images. Fig.

13 shows that the results of the color histogram equalization and the proposed method are acceptable, whereas the result of the multi-scale retinex with color restoration has many noises and strong halo effects. As shown in Fig. 14, although all the results are acceptable, our result is shown to be greenish on the whole. Table 1 and Table 2 show values of camera setting used to obtain test images.

From the experimental results, it has been shown that the proposed method achieves better color reproduction of the test images taken under various environments, compared with the existing methods.

In addition, in order to improve the computation time of our method, we made 8448 bytes look-up table (with 256 input cases x 3 color channels x 11 EV). The look-up table includes a set of all cases of input and output pixel value pairs. For the 5184x3456 size images, the proposed method reproduced the colors in about 0.6 second in average using a desktop computer consists of Intel Core 2 Duo 2.93 GHz and 4 GB DDR RAM. For this reason, the proposed method can be performed in real-time.

(9)

Fig. 7. Experimental results of the proposed method for various EV (a) image taken with proper EV indicated by an exposure meter in the digital camera (b), (c), (d), and (e) are images taken with +1, +2, +3, and +4 EV from the proper EV, respectively. (f), (g), (h), and (i) are reproduced images of (b), (c),

(d), and (e), respectively.

Fig. 8. Experimental results of the proposed method for various EV (a) image taken with proper EV indicated by an exposure meter in the digital camera (b), (c), (d), and (e) are images taken with +1, +2, +3, and +4 EV from the proper EV, respectively. (f), (g), (h), and (i) are reproduced images of (b), (c),

(d), and (e), respectively.

(10)

Fig. 9. Experimental comparison of the proposed method with the existing methods (a) image taken with proper EV indicated by an exposure meter in the digital camera (b) is an image taken with +4 EV

from the proper EV. (c) is a result image of (b) by using color histogram equalization method. (d) is a result image of (b) by using multi-scale retinex with color restoration method. (e) is a result image of (b)

by using the proposed method.

(11)

(12)

Fig. 13. Experimental comparison of the proposed method with the existing methods (a) image taken with proper EV indicated by an exposure meter in the digital camera (b) is an image taken with +4 EV from the proper EV. (c) is a result image of (b) by using color histogram equalization method. (d) is a result image of (b) by using multi-scale retinex with color restoration method. (e) is a result image of (b) by using the proposed method.

Fig. 14. Experimental comparison of the proposed method with the existing methods (a) image taken with proper EV indicated by an exposure meter in the digital camera (b) is an image taken with +4 EV from the proper EV. (c) is a result image of (b) by using color histogram equalization method. (d) is a result image of (b) by using multi-scale retinex with color restoration method. (e) is a result image of (b) by using the proposed method.

(13)

Table 1. Values of camera setting used in Fig. 7 and Fig. 8.

Settings Fig. 7 Fig. 8 (a)

Focal length 35mm 28mm

f-number f/5.6 f/5.6

Shutter speed 1/8s 1/4s (b)

Shutter speed 1/15s 1/8s (c)

Shutter speed 1/30s 1/15s (d)

Shutter speed 1/60s 1/30s (e)

Shutter speed 1/125s 1/60s

Table 2. Values of camera setting used from Fig. 9 to Fig. 14.

Settings Fig. 9 Fig. 10 Fig. 11 Fig. 12 Fig. 13 Fig. 14 (a)

Focal length 35mm 28mm 34mm 55mm 28mm 34mm

f-number f/5.6 f/5.6 f/5.6 f/5.6 f/5.6 f/5.6

Shutter speed 1/8s 1/4s 1/10s 1/4s 1/8s 1/45s

(b)

Focal length 35mm 28mm 34mm 55mm 28mm 34mm

f-number f/5.6 f/5.6 f/5.6 f/5.6 f/5.6 f/5.6

Shutter speed 1/125s 1/60s 1/180s 1/60s 1/125s 1/750s

5. Conclusion

In this paper, we proposed a new technique for color reproduction based on color variation characteristics of digital camera under dim light conditions. Under-exposed input images can be taken automatically by digital camera with above a safe shutter speed to prevent its blur effect under a dim light condition. Then the non-blurred output image with the proper exposure and the color can be obtained by using the proposed method. The experimental results show that the color and the tone of output images are most similar to the ones of images taken as proper EV, compared with the existing methods. The proposed method can be performed in real-time, because the color variation characteristics of all cases for enhancing the input pixels to the output pixels are calculated at off-line step. Moreover, since the proposed method uses camera EV indicated by camera exposure meter, the proposed method can be properly used as camera built-in algorithm. Therefore, photographers can effectively take a picture regardless of the blurred image caused by their shaking hands under a dim light condition.

References

[1] G. Petschnigg, R. Szeliski, M. Agrawala, M. Cohen, H. Hoppe, and K. Toyama, “Digital Photography with Flash and No-Flash Image Pairs,” ACM Trans. Graphics, vol. 23, no. 3, pp.

664-672, Aug. 2004. Article (CrossRef Link)

(14)

[2] A. Adams, E. Talvala, S.H. Park, D.E. Jacobs, B. Ajdin, N. Gelfand, J. Dolson, D. Vaquero, J.

Baek, M. Tico, H.P. A. Lensch, W. Matusik, K. Pulli, M. Horowitz, M. Levoy, “The Fankencamera: An Experimental Platform for Computational Photography,” ACM Trans.

Graphics, vol. 29, no. 4, pp. 1-12, July 2010. Article (CrossRef Link)

[3] E. Reinhard, G. Ward, S. Pattanaik, P. Debevec, “High dynamic range imaging: acquisition, display and image-based lighting,” Morgan Kaufmann , 2005.

[4] Y. Monobe, H. Yamashita, T. Kurosawa, H. Kotera, “Dynamic Range Compression Preserving Local Image Contrast for Digital Video Camera,” IEEE Trans. Consumer Electronics, vol. 51, no.

1, pp. 1-10, Feb. 2005. Article (CrossRef Link)

[5] W. Kao, L. Chen, S. Wang, “Tone Reproduction in Color Imaging Systems by Histogram Equalization of Macro Edges,” IEEE Trans. Consumer Electronics, vol. 52, no. 2, pp. 682-688, May 2006. Article (CrossRef Link)

[6] R. Fergus, B. Singh, A. Hertzmann, S. T. Roweis, W. T. Freeman, “Removing Camera Shake from a Single Photograph,” ACM Trans. Graphics, vol. 25, no. 3, pp. 787-794, July 2006. Article (CrossRef Link)

[7] L. Yuan, J. Sun, L. Quan, H.-Y. Shum, “Image Deblurring with Blurred/Noisy Image Pairs,” ACM Trans. Graphics, vol. 26, no. 3, pp. 1-10, July 2007. Article (CrossRef Link)

[8] N. Bassiou, C. Kotropoulos, “Color Image Histogram Equalization by Absolute Discounting Back-Off,” Computer Vision and Image Understanding, vol. 107, no. 1-2, pp. 108-122, 2007.

Article (CrossRef Link)

[9] E. Land, “An Alternative Technique for the Computation of the Designator in the Retinex Theory of Color Vision,” in Proc. of the National Academy Science, vol. 83, no. 1-2, pp. 3078-3080, 1986.

Article (CrossRef Link)

[10] D.J. Jobson, Z. Rahman, G.A. Woodell, “Properties and Performance of a Center/Surround Retinex,” IEEE Trans. Image Processing, vol. 6, no. 3, pp. 451-462, Mar. 1997. Article (CrossRef Link)

[11] R. Ng, M. Levoy, M. Bredif, G. Duval, M. Horowitz, P. Hanrahan, “Light Field Photography with a Hand-Held Plenoptic Camera,” Tech Report CTSR 2005-02, Computer Science Department of Stanford University, 2005.

[12] S. Bae, F. Durand, “Defocus Magnification,” Computer Graphics Forum, vol. 26, no. 3, pp.

571-579, 2007. Article (CrossRef Link)

[13] J.H. Lee, K.T. Park, Y.S. Moon, “Image Deblurring by Using the Estimation of PSF Parameters for Image Devices,” in Proc. IEEE Conf. Consumer Electronics, pp. 387-388, January, 2010. Article (CrossRef Link)

[14] J. Jia, J. Sun, C.-K. Tang, H.-Y. Shum, “Bayesian Correction of Image Intensity with Spatial Consideration,” in Proc. of ECCV 2004, Lecture Notes in Computer Science, vol. 3023, pp.

342-354, 2004. Article (CrossRef Link)

[15] B. Peterson, “Understanding Exposure,” 3rd ed., Amphoto Books, 2010.

[16] H. Lee, “Introduction to Color Imaging Science,” Cambridge University Press, 2009.

[17] M. Langford, “Basic Photography,” 7th ed., Focal Press, 2000.

[18] B. Long, “Complete digital photography,” 4th ed., Charles River Media, 2007.

(15)

Ui Seong Kim received his B.S. degree in computer science and engineering from Hanyang University, Republic of Korea, in 2010. He is currently working toward his M.S. degree at the computer vision and pattern recognition laboratory, Department of Computer Science and Engineering, Hanyang University, Republic of Korea. His major research interests are in the areas of computational photography, image enhancement, digital signal processing.

Jong Min Lee received his B.S. and M.S. degrees in computer science and engineering from Hanyang University, Republic of Korea, in 2007 and 2009, respectively. He is currently working toward his Ph.D. degree at the computer vision and pattern recognition laboratory, Department of Computer Science and Engineering, Hanyang University, Republic of Korea. His major interests include image enhancement, object detection, and target tracking.

Yong Min Kim received his M.S. degree in computer science and engineering from Hanyang University, Republic of Korea, in 2011. He is currently working toward his Ph.D. degree at the computer vision and pattern recognition laboratory, Department of Computer Science and Engineering, Hanyang University, Republic of Korea. His major interests include face processing, object segmentation, image enhancement, and augmented reality.

Ki Tae Park received the B.S., M.S., and Ph.D. degrees in computer science and engineering from Hanyang University, Republic of Korea, in 2000, 2002, and 2007, respectively. He had worked in Samsung Electronics Co., Ltd., Suwon, Republic of Korea and worked as research professor at Hanyang University. He has been an assistant professor in General Education Curriculum Center at Hanyang University. His major research interests are in the areas of digital signal processing, content-based image retrieval, object segmentation, multimedia processing, and video indexing.

Young Shik Moon is the corresponding author of this paper. He received the B.S. and M.S. degrees in electronics engineering from Seoul National University and Korea Advanced Institute of Science and Technology (KAIST), Republic of Korea, in 1980 and 1982, respectively. From 1982 to 1985, He had been a researcher at Electronics and Telecommunications Research Institute, Republic of Korea. He received the Ph.D. in computer engineering from University of California at Irvine, CA, in 1990. Since 1992, he has been a professor in Department of Computer Science and Engineering at Hanyang University, Republic of Korea. His research interests include computer vision, image processing, computer graphics, and computational photography.