Camera Calibration

Camera Calibration

Wanho Choi

(wanochoi.com)

• How to solve Ax=b : http://wanochoi.com/lecture/Ax=b.pdf • Least Squares : http://wanochoi.com/lecture/Least_Squares.pdf • Transformation Matrix : http://wanochoi.com/lecture/TransformationMatrix.pdf • Rotation : http://wanochoi.com/lecture/Rotation.pdf

Preliminaries

How to capture? [Idea #1]

film

(photon sensor)

the captured image

no image fully blurred

object

the sun

How to capture? [Idea #2]

the captured image object film barrier a small hole (known as aperture) (photon sensor) upside-down sharp

but, insufficient light

requires large exposure time

the sun

How to capture? [Idea #3]

film

lens (photon sensor)

the captured image

upside-down sharp sufficient light object the sun (light source)

Convex Lens Formula

Aperture (

조리개)

• 렌즈(lens)로 들어오는 광량(amount of light)을 조절한다.

• 또한, 피사계 심도(depth-of-field)도 이에 영향을 받는다.

: 초점이 맞은 것으로 인식되는 (acceptably sharp) 거리의 범위

• 구멍의 크기가 작을수록 핀홀 카메라(pinhole camera)에 가까워 진다.

Aperture (

조리개)

Aperture (

조리개)

Ideal Thin Lens vs Real Lens

spherical aberration chromatic aberration

https://expertphotography.com/chromatic-aberration-photography/ https://en.wikipedia.org/wiki/Spherical_aberration http://www.drewgrayphoto.com/learn/distortion101

• Pinhole as a point

• Pencil of rays: 모든 광선(ray)은 한 점(focal point)을 통과

• One ray per each point

Pinhole Camera

• “어두운 방”이라는 뜻

• 그림 등을 그리기 위해 만든 광학 장치로, 사진술의 전신

Camera Obscura

Camera Obscura

Homogeneous Coordinate System

• 유클리디언 기하학(Euclidean geometry)에서 사용하는 좌표계 : 데카르트 좌표계 (cartesian coordinate system)

• _{사영 기하학(projective geometry)에서 사용하는 좌표계}

: 동차 좌표계 (homogeneous coordinate system)

(x, y, z) (x, y, z,1) (x, y, z, w)

→

→

→

→

Homogeneous Coordinate

• The 2D point (x, y) is represented by the homogeneous coordinate (x, y,1).

• In general, the homogeneous coordinate (x, y, w) represents the 2D point (x/w, y/w).

Pinhole Camera Model

world space (3D) ➔ camera space (3D)

image plane space (2D) ➔ pixel space (sensor) (2D)

y z O = C0 : optical center : optical axis P(XC, YC, ZC) p = ? f: focal length image plane p = [xy] ≡ ˜x ˜y ˜z = f XC/ZC f YC/ZC 1 ≡ f Xf YCC ZC = 0 f 0 0f 0 0 0 0 0 1 0 XC YC ZC 1 pixel scale image center skewness

(usually negligible or zero)

rotation translation projection ˜x ˜y ˜z = f 0 0 0 0 f 0 0 0 0 1 0 X_C YC Z_C 1 f 0 0 0 f 0 0 0 1 [ 1 0 0 0 0 1 0 0 0 0 1 0] zooming 3D to 2D

: standard (or canonical) projection matrix

Pinhole Camera Model

˜u

˜v

˜

w

=

s

s

u

0 s

v

0 0 1

f 0 0 0

0 f 0 0

0 0 1 0

r

r

r

t

r

r

r

t

r

r

r

t

0 0 0 1

X

Y

Z

1

Pinhole Camera Model

˜u

˜v

˜

w

=

s

s

u

0 s

v

0 0 1

f 0 0 0

0 f 0 0

0 0 1 0

r

r

r

t

r

r

r

t

r

r

r

t

0 0 0 1

X

Y

Z

1

K: intrinsic parameters W: extrinsic parameters

Pinhole Camera Model

˜u

˜v

˜

w

=

s

s

u

0 s

v

0 0 1

f 0 0 0

0 f 0 0

0 0 1 0

r

r

r

t

r

r

r

t

r

r

r

t

0 0 0 1

X

Y

Z

1

K: intrinsic parameters W: extrinsic parameters

C: camera matrix

K =

α γ u

0 β v

0 0 1

=

s

s

u

0 s

v

0 0 1

f 0 0 0

0 f 0 0

0 0 1 0

Homography

• Transformation between two different planes

• Homography matrix

‣ 3x3 square matrix

‣ But, 8 DoF as it is estimated up to a scale

‣ It is generally with . s x′y′ 1 = H [ x y 1] = h₁₁ h₁₂ h₁₃ h₂₁ h₂₂ h₂₃ h₃₁ h₃₂ h₃₃ [ x y 1] h₃₃ = 1

The Examples of Homography

DLT (Direct Linear Transformation)

˜u

˜v

˜

w

=

α γ u

0 β v

0 0 1

r

r

r

t

r

r

r

t

r

r

r

t

0 0 0 1

X

Y

Z

1

DLT (Direct Linear Transformation)

˜u

˜v

˜

w

=

α γ u

0 β v

0 0 1

r

r

r

t

r

r

r

t

r

r

r

t

0 0 0 1

X

Y

Z

1

p = CP

DLT (Direct Linear Transformation)

p = CP

˜u

˜v

˜

w

=

C

C

C

C

C

C

C

C

C

C

C

C

X

Y

Z

1

DLT (Direct Linear Transformation)

DLT (Direct Linear Transformation)

• For N-point pairs

−X₁ −Y₁ −Z₁ −1 0 0 0 0 u₁X₁ u₁Y₁ u₁Z₁ u₁ 0 0 0 0 −X₁ −Y₁ −Z₁ −1 v₁X₁ v₁Y₁ v₁Z₁ v₁ −X₂ −Y₂ −Z₂ −1 0 0 0 0 u₂X₂ u₂Y₂ u₂Z₂ u₂ 0 0 0 0 −X₂ −Y₂ −Z₂ −1 v₂X₂ v₂Y₂ v₂Z₂ v₂ ⋮ ⋮ −X_N −Y_N −Z_N −1 0 0 0 0 u_NX_N u_NY_N u_NZ_N u_N 0 0 0 0 −X_N −Y_N −Z_N −1 v_NX_N v_NY_N v_NZ_N v_N C₁₁ C₁₂ C₁₃ C₁₄ C₂₁ C₂₂ C₂₃ C₂₄ C₃₁ C₃₂ C₃₃ C₃₄ = 0 12N × 12 12 × 1

Mc = 0

DLT (Direct Linear Transformation)

Mc = 0 Mc = w

!

̂c = argmin

c

(w

T

_w)

= cT_(VSUT_{) (USV}T_{) c = c}T_VSUT_USVT_c

= cTVS2VTc = cT ( 12 ∑ i=1 s_i2v_iv_iT ) c

: SVD (Singular Value Decomposition)

: the 12th (=smallest) eigenvector of V

= cT _(s2

1v1v1T + s22v2v2T + ⋯ + s122 v12v12T ) c

∴ ̂c = v_{12 ( ∵ v}T_i v_{j = 0)}

If individual parameters are needed

C = [H|h] = KR [I| − C

] = H [I| − C

]

H = QR

∴ K = 1

Q

Q

h = − HC

∴ C

= − Hh

• Checkerboard

‣ Size & structure are known.

‣ Easy to set & get the points.

Camera Calibration using 2D Pattern

• All points are on a plane, so Z=0.

x

y

z

Camera Calibration using 2D Pattern

• All points are on a plane, so Z=0.

• We cannot solve the problem with general DLT process.

−X₁ −Y₁ −Z₁ −1 0 0 0 0 u₁X₁ u₁Y₁ u₁Z₁ u₁ 0 0 0 0 −X₁ −Y₁ −Z₁ −1 v₁X₁ v₁Y₁ v₁Z₁ v₁ −X₂ −Y₂ −Z₂ −1 0 0 0 0 u₂X₂ u₂Y₂ u₂Z₂ u₂ 0 0 0 0 −X₂ −Y₂ −Z₂ −1 v₂X₂ v₂Y₂ v₂Z₂ v₂ ⋮ ⋮ −X_N −Y_N −Z_N −1 0 0 0 0 u_NX_N u_NY_N u_NZ_N u_N 0 0 0 0 −X_N −Y_N −Z_N −1 v_NX_N v_NY_N v_NZ_N v_N C₁₁ C₁₂ C₁₃ C₁₄ C₂₁ C₂₂ C₂₃ C₂₄ C₃₁ C₃₂ C₃₃ C₃₄ = 0 rank deficiency!

A Simple Trick!

˜u

˜v

˜

w

=

α γ u

0 β v

0 0 1

r

r

r

t

r

r

r

t

r

r

r

t

0 0 0 1

X

Y

Z

1

˜u

˜v

˜

w

=

α γ u

0 β v

0 0 1

r

r

t

r

r

t

r

r

t

[

X

Y

1]

H = [h

, h

, h

] = K[r

, r

, t]

Homography

• Linear transformation between two different planes

p = HP

[

u

v

1]

= [h

, h

, h

] [

X

Y

1]

[

u

v

1]

=

h

h

h

h

h

h

h

h

h

[

X

Y

1]

= 1

So, we need at least 4 point pairs.

How to get K, R, and T from H

H = [h

, h

, h

] = K[r

, r

, t]

r

= K

h

r

= K

h

r

r

= 0

r

r

= r

r

= 1

∴

How to get K, R, and T

hT₁K−TK−1h₂ = 0

hT

1K−TK−1h1 − hT2K−TK−1h2 = 0

: symmetric positive definite matrix

B = b₁₁ b₁₂ b₁₃ b₂₁ b₂₂ b₂₃ b₃₁ b₃₂ b₃₃ ∵ B = K−T_K−1 _{= (K}−T_{) (K}−T₎T _{= AA}T _{: Cholesky decomposition} ∴ b := [b11 b12 b13 b22 b23 b33]T B := K−T_K−1 h T 1Bh2 = 0 hT 1Bh1 − hT2Bh2 = 0

How to get K, R, and T

How to get K, R, and T

the 1st point pair

the n-th point pair

vT₁₂ (v11 − v12)T ⋯ vT 12 (v11 − v12)T b = 0

Vb = 0

Vb = 0 Vb = w

!

̂b = argmin

b

(w

T

_w)

→

→

If individual parameters are needed

H = [h

, h

, h

] = K[r

, r

, t]

r

= K

h

r

= K

h

B = K

_K

B = K

K

= AA

∴ K = A

r

= K

_h

&

r

= K

h

h

= Kt → t = K

h

• Distortion: non-linear error

• Especially radial distortion

• So, we only consider the first two terms of radial distortion.

• The distortion function is dominated by the radial components, and especially dominated by the first term.

• Moreover, more elaborated model would cause numerical instability.

Dealing with Radial Distortion

˘x = x + x [k1(x2 + y2)2+k2(x2 + y2)2]

˘y = y + y [k1(x2 + y2)2+k2(x2 + y2)2] (˘x, ˘y)

(x, y): ideal (distortion-free) point : real (distorted) point

˜u ˜v 1 = α 0 u_c 0 β v_c 0 0 1 ˜x ˜y 1 ˜u = u_c + α ˘x ˜v = v_c + α˘y ˘u = u + (u − u0) x [k1(x2 + y2)2+k2(x2 + y2)2] ˘v = v + (v − v0) x [k1(x2 + y2)2+k2(x2 + y2)2] [(u − u 0) (x2 + y2) (u − u0) (x2 + y2) (v − v0) (x2 + y2) (v − v0) (x2 + y2)] [ k₁ k₂] = [˘v − v]˘u − u

Maximum Likelihood Estimation

• Non-linear optimization problem

• Levenberg-Marquardt algorithm

• Initial guess from DLT

N

∑

∑

p

− ˘p

_{(K, k}

, k

, R

, t

, P

₎

Camera Calibration using OpenCV

References